[semantics-public] [x3d-public] x3d.py package: Some feedback and suggestion for improvement; tuples and lists; updated x3d.py
John Carlson
yottzumm at gmail.com
Wed Feb 23 16:07:23 PST 2022
I have also run into issues with field and fieldValue as well in
x3djsonld.py. Solutions are still on workbench, though.
I discovered that ParticleSystem is a subclass of X3DShapeNode (?).
I’m looking forward to the new release, but I am relaxing now.
I continue to think about how we can avoid large amounts of XSLT or
slightly modified JS when we add new bindings and encodings. DFDL seems
to have a 1-1 approach. I am not sure what Altova might provide.
Perhaps we need to build abstract and concrete language models? Indeed,
what is a language model?
Hmm
John
On Wed, Feb 23, 2022 at 9:30 AM Brutzman, Donald (Don) (CIV) <
brutzman at nps.edu> wrote:
> Have released another build of x3d.py (aka X3DJSAIL), now version 4.0.51.
>
> - https://pypi.org/project/x3d
> - https://www.web3d.org/x3d/stylesheets/python/python.html
>
>
>
> Updated features include:
>
> - Override (and report) when a contained MetadataSet (or other
> Metadata*) node has incorrect containerField, providing extra care since
> default values changed in X3D4. Similar checks are provided in X3dTidy
> stylesheet.
> - When a conversion error to XML (.x3d output file) occurs, only print
> out 7 lines of result file containing the error (instead of the whole file).
> - Proper handling of typed values contained by <field> or <fieldValue>
> initializations for Script nodes and prototypes.
>
>
>
> create python:
>
> C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors/Chapter14Prototypes//TextStringPrototype.x3d
> processing with X3dToPython stylesheet...
>
>
>
> C:\x3d-code\www.web3d.org\x3d\content\examples\X3dForWebAuthors/Chapter14Prototypes//TextStringPrototype.py
> self-validation tests...
>
> validate python:
>
> x3d.py package loaded, have fun with X3D Graphics!
>
> Python-to-XML well-formed XML document test of XML output complete
>
> Python-to-XML X3D 3.3 schema validation test of XML output complete
>
> Python-to-VRML export of VRML output successful
>
> Python-to-JSON export of JSON output successful (under development)
>
> python x3d.py load and self-test complete for TextStringPrototype.py
>
>
>
> -
> https://x3dgraphics.com/examples/X3dForWebAuthors/Chapter14Prototypes/TextStringPrototypeIndex.html
> -
> https://x3dgraphics.com/examples/X3dForWebAuthors/Chapter14Prototypes/TextStringPrototype.py
> -
> https://x3dgraphics.com/examples/X3dForWebAuthors/Chapter14Prototypes/TextStringPrototype.x3d
>
>
>
>
>
> The regression tests are running very cleanly, when processing the X3D
> Examples archives a handful of errors in field initializations were
> detected and now few exceptions are reported. Perhaps the only recurring
> exceptions are when some of our extremely large scans are encountered, too
> big for my laptop to published.
>
>
>
> - https://www.web3d.org/x3d/content/examples/build.python.all.log.txt
>
>
>
> Also upgraded the Saxon xslt engine to version 11.2 of open-source home
> edition (HE), to good effect.
>
>
>
> - https://www.saxonica.com/html/documentation11/about
>
>
>
> Have fun with X3D Python! 8)
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, Navy robotics https://
> faculty.nps.edu/brutzman
>
>
>
> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
> *Sent:* Monday, February 21, 2022 9:41 AM
> *To:* Hans Moritz Guenther <hgunther at mit.edu>; X3D Public Mailing List (
> x3d-public at web3d.org) <x3d-public at web3d.org>
> *Cc:* rlentz <rlentz at gmail.com>; Vince Marchetti <vmarchetti at kshell.com>;
> Mike McCann <mccann at mbari.org>; Peitso, Loren (CIV) <lepeitso at nps.edu>;
> Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
> *Subject:* RE: x3d.py package: Some feedback and suggestion for
> improvement; tuples and lists; updated x3d.py
>
>
>
> Hello all python programmers. Another x3d.py release is available. Have
> performed a complete refactoring of field type setters as well as
> corresponding isValid* and assertValid* methods. The field types now
> perform value validation whenever setting, and duck typing is preserved
> wherever possible up and down within the X3D field types. Now published to
> PyPi with corresponding unit tests complete.
>
> - x3d 4.0.50
> - pip install x3d==4.0.50
> - https://pypi.org/project/x3d/4.0.50
> - X3D Examples Archive regression tests, several post-test
> improvements are being added:
> - https://www.web3d.org/x3d/content/examples/build.python.all.log.txt
>
>
>
> Once again a large number of smoke tests have been added, confirming it is
> indeed a good way to check new functionality.
>
> -
> https://www.web3d.org/x3d/stylesheets/python/examples/PythonX3dSmokeTests.py
> - https://www.web3d.org/x3d/stylesheets/python/build.examples.log.txt
>
>
>
> I’ve looked at numpy a little, but there are many types. We’re ready to
> consider utility methods. Up until now, I have hesitated to offer
> plain-array support for data types since that might easily break tuple
> checks.
>
>
>
> Example(s) showing common use cases for X3D with numpy would be helpful,
> TIA.
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, Navy robotics https://
> faculty.nps.edu/brutzman
>
>
>
> *From:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>
> *Sent:* Sunday, February 13, 2022 11:08 PM
> *To:* Hans Moritz Guenther <hgunther at mit.edu>; Peitso, Loren (CIV) <
> lepeitso at nps.edu>
> *Cc:* rlentz <rlentz at gmail.com>; Vince Marchetti <vmarchetti at kshell.com>;
> Mike McCann <mccann at mbari.org>; X3D Public Mailing List (
> x3d-public at web3d.org) <x3d-public at web3d.org>; Brutzman, Donald (Don)
> (CIV) <brutzman at nps.edu>
> *Subject:* RE: x3d.py package: Some feedback and suggestion for
> improvement; tuples and lists; updated x3d.py
>
>
>
> I have updated x3d.py to include several points of progress.
>
>
>
> 1. REGEX_PYTHON (perturbated from REGEX_XML) for all types.
> Validation of each X3D-typed python value now includes regex checking.
> 2. The standard python library ‘re’ for regexes is now imported by
> x3d.py
> 3. In X3D XML Schemas and X3DUOM, made regex occurrence (\s)
> consistent throughout as \s
> 4. Strings are now allowed as validatable values for some simple
> types. More work needed for list and array combinations.
> 5. Documentation and tests:
>
>
> - Python X3D Package x3d.py
> - “The x3d.py Python X3D Package supports programmers with Python
> interfaces and objects for standards-based X3D programming, all as open
> source.”
> - https://www.web3d.org/x3d/stylesheets/python/python.html
>
>
>
> 1. A large number of smoke tests have been added, this is indeed a
> good way to check new functionality.
>
>
> -
> https://www.web3d.org/x3d/stylesheets/python/examples/PythonX3dSmokeTests.py
> - https://www.web3d.org/x3d/stylesheets/python/build.examples.log.txt
>
>
>
> 1. Have added section on Data Types to X3D Python page.
>
>
>
> - https://www.web3d.org/x3d/stylesheets/python/python.html#DataTypes
>
>
>
> *🔖 <https://www.web3d.org/x3d/stylesheets/python/python.html#DataTypes>
> Data Types *
>
> An important aspect of Python ease of programming is duck typing
> <https://en.wikipedia.org/wiki/Duck_typing>: If it looks like a duck and
> walks like a duck and quacks like a duck, then it must be a duck.
>
> Python duck typing
> <https://docs.python.org/3/glossary.html#term-duck-typing> describes how
> well-designed code improves its flexibility by allowing polymorphic
> substitution (i.e. parametric polymorphism
> <https://en.wikipedia.org/wiki/Parametric_polymorphism>).
>
> X3D has strictly defined field types
> <https://www.web3d.org/specifications/X3Dv4Draft/ISO-IEC19775-1v4-CD1/Part01/fieldTypes.html>
> for every field of every node in the standard. This means that all values
> can be statically checked when defining a model, leading to high-quality
> model files. Further it is an error to provide an incorrectly typed value
> at run time, i.e. an SFString cannot be directly used as a numeric value.
> This approach precludes a wide range of errors that can occur in other
> languages. Details on each possible X3D type can be found at X3D Tooltips:
> types <https://www.web3d.org/x3d/content/X3dTooltips.html#type>.
>
> Strict typing checks are included when building X3D classes and assigning
> values to fields, for all X3D types. This strictness for input data
> prevents Garbage in, garbage out
> <https://en.wikipedia.org/wiki/Garbage_in,_garbage_out> pathologies when
> creating X3D models.
>
> X3D regular expression (regex)
> <https://www.web3d.org/specifications/X3dRegularExpressions.html>
> checking is applied to result values. Each type class includes string
> constants for REGEX_PYTHON and REGEX_PYTHON patterns.
>
> Python authors can confirm data validity and take further advantage of
> available type-checking methods, such as the following:
>
> - Checks *value* expression and reports results as boolean:
> isValidSFBool(*value*), isValidSFVec3f(*value*), isValidMFRotation(
> *value*), etc.
> - Checks *value* expression and assets valid results, or else Fails
> and throws traceback:
> assertValidSFBool(*value*), assertValidSFVec3f(*value*),
> assertValidMFRotation(*value*), etc.
>
> The x3d.py python package strives to support duck typing of unambiguous
> values while preserving strict X3D validation of all inputs. Example tests
> for each case can be found in examples/PythonX3dSmokeTests.py
> <https://www.web3d.org/x3d/stylesheets/python/examples/PythonX3dSmokeTests.py>
> with corresponding outputs shown in build.examples.log.txt
> <https://www.web3d.org/x3d/stylesheets/python/build.examples.log.txt>
>
> 1. Update available for installation/refresh
>
>
> - x3d 4.0.49
> - pip install x3d==4.0.49
> - https://pypi.org/project/x3d/4.0.49
>
> g. It would be helpful to have a simple example program contributed that
> uses both x3d.py and numpy packages, this will doubtless inform current
> development future testing.
>
> Unit tests appear to be passing, I hope these changes are indeed
> improvements. Updated test reports welcome.
>
>
>
> Have fun with X3D Python! 8)
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, Navy robotics https://
> faculty.nps.edu/brutzman
>
>
>
> *From:* Brutzman, Donald (Don) (CIV)
> *Sent:* Saturday, February 12, 2022 5:50 PM
> *To:* Hans Moritz Guenther <hgunther at mit.edu>; Peitso, Loren (CIV) <
> lepeitso at nps.edu>
> *Cc:* rlentz <rlentz at gmail.com>; Vince Marchetti <vmarchetti at kshell.com>;
> Mike McCann <mccann at mbari.org>; X3D Public Mailing List (
> x3d-public at web3d.org) <x3d-public at web3d.org>; brutzman at nps.edu
> *Subject:* RE: x3d.py package: Some feedback and suggestion for
> improvement; tuples and lists
>
>
>
> [Cc: x3d-public list, all design discussions are open and archived]
>
>
>
> Splitting out one issue at a time (see subject line please)
>
>
>
> Moritz wrote:
>
> This works:
>
> x3d.Coordinate(point=[[0, 0, 0], (0, 1, 0), (1, 1, 0)])
>
> This fails:
>
> x3d.Coordinate(point=([0, 0, 0], (0, 1, 0), (1, 1, 0)))
>
> X3DTypeError: ([0, 0, 0], (0, 1, 0), (1, 1, 0)), type=<class 'tuple'> is not a valid Python list for MFVec3f
>
>
>
> Why not allow a tuple instead of a list here? The inner coordinates can be either tuples or lists.
>
>
>
> I don't understand what that typechecking is good for here.
>
>
>
> Background: X3D types include strongly typed tuples (Single Field Vector, SFVec* types) and ordered lists of those same types (Multiple Field Vector, MFVec* types)
>
>
>
> · X3D4 Architecture, clause 5 Field type reference
>
> · https://www.web3d.org/specifications/X3Dv4Draft/ISO-IEC19775-1v4-CD1/Part01/fieldTypes.html
>
>
>
> further explained at
>
> · X3D Tooltips, Type definitions
>
> · https://www.web3d.org/x3d/content/X3dTooltips.html#type
>
>
>
> Not yet described in the documentation, but evident in the code, is the design decision to use python tuples for SFVec types, and python lists for MFVec types.
>
>
>
> This remains the most logical alignment with Python data structures.
>
>
>
> Glad to see your test case provoked an understandable error! Indeed
>
>
>
> The intent of strictness is to avoid run-time problems that can easily arise from lackadaisical treatment of arrays. Something like your example might look obvious in an example composed of numeric data, but might be much less clear if variables are used.
>
>
>
> Am further concerned that if mixed combinations of values are allowed, then isolation of errors within long lists might not be possible/practical if error messages remain as terse as above (or, conversely, if error messages are immensely complex).
>
>
>
> If is conceivable we might find some cases where we can safely add support. For example, creating a new SFVec3f([1,2,3]) might be allowable.
>
>
>
> Am thinking we should be careful on this one. Defining some simple acceptable variations that we might add to the Smoke Tests would be a good next step.
>
>
>
> · https://www.web3d.org/x3d/stylesheets/python/examples/PythonX3dSmokeTests.py
>
> · https://www.web3d.org/x3d/stylesheets/python/build.examples.log.txt
>
>
>
> The place to put any relaxations will be in the type classes themselves,
> e.g. SFVec3f and MFVec3f etc.
>
>
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br
> brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA
> +1.831.656.2149
>
> X3D graphics, virtual worlds, Navy robotics https://
> faculty.nps.edu/brutzman
>
>
>
> *From:* Hans Moritz Guenther <hgunther at mit.edu>
> *Sent:* Friday, February 11, 2022 7:45 PM
> *To:* Peitso, Loren (CIV) <lepeitso at nps.edu>
> *Cc:* Brutzman, Donald (Don) (CIV) <brutzman at nps.edu>; rlentz <
> rlentz at gmail.com>; Vince Marchetti <vmarchetti at kshell.com>; Mike McCann <
> mccann at mbari.org>
> *Subject:* Re: x3d.py package: Some feedback and suggestion for
> improvement
>
>
>
> On 2/11/22 8:47 PM, Peitso, Loren (CIV) wrote:
>
> Hans,
>
>
>
> I have had a chat with Don and we are in full agreement that your
> described use case is important and would be quite useful for widening the
> usefulness of the x3d package. Consideration for expanding usability is to
> not break anything (obviously), remain Pythonic, and not add additional
> code constructs/constraints that require consideration of architectural
> changes every time something needs adding or updating. I'll propose a
> couple solutions by issue-raised and we can discuss debate from there.
>
>
>
> " I need to use "x3d.x3d". That's of course possible but a little
> cumbersome."
>
> Agreed on cumbersome! Have you tried import x3d.x3d as x3d_ ?
>
> Sure (I even use import x3d.x3d as x3d). But it's not how most python
> packages work, stating with the standard library:
>
> from os import chdir, not from os.os import chdir
>
> from math import atanh, not from math.math import atanh
>
> The same is true for common Python packages that are not part of the
> standard library:
>
> import numpy as np; arr = np.ones(5), not import numpy.numpy as np
>
> import pandas as pd, not import pandas.pandas as pd
>
> It is of course your prerogative as package author to chose any naming
> scheme that seems sensible to you, but it is certainly unusual to import a
> package this way. It *does* make sense if you plan for more functionality
> in the future, e.g. if you think there could be other submodules such as
>
> x3d.json
>
> x3d.numpy_interface
>
> x3d.utils
>
> x3d.display
>
> etc.
>
> If that's the case, maybe a short comment in the code that explains the
> reasoning would be useful.
>
>
>
>
>
> - the __init__.py file. The file is present, but not correct. It lists all
> the classes in __all__ but does not actually import them. Thus, that can't
> actually be used.
>
> Per https://docs.python.org/3/tutorial/modules.html
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Ftutorial%2Fmodules.html&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006429169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=mTmjQB20FGw1meaCSQECcfGpR2KghiV9IH56m7ERT%2BU%3D&reserved=0>,
> section 6.4:
>
> "In the simplest case, __init__.py can just be an empty file, but it can
> also execute initialization code for the package or set the __all__variable,
> described later."
>
>
>
> Per https://docs.python.org/3/tutorial/modules.html
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Ftutorial%2Fmodules.html&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=%2FmuyUTirss61Zu3fe1ExoeuIk1%2F7ou5SfuhFnlc9cC0%3D&reserved=0>,
> section 6.4.1:
>
> "Although certain modules are designed to export only names that follow
> certain patterns when you use import *, it is still considered bad
> practice in production code."
>
>
>
> The "not actually import(ing) them part" is working as Python intends when
> we provide the empty __init__.py file (per 6.4.1)
>
> If __all__ is not defined, the statement from sound.effects import *does
> *not* import all submodules from the package sound.effects into the
> current namespace; it only ensures that the package sound.effects has
> been imported (possibly running any initialization code in __init__.py)
> and then imports whatever names are defined in the package.
>
> The __all__ attribute gets initialized during the importing, but only as
> far as described. Thus requiring the package name prepend for
> fully-qualified names to properly resolve.
>
> We have chosen the empty __init__.py file convention for simplicity, as
> there is no forced loading of any leading underscore names into the _all__
> attribute and/or namespace by using this convention (respecting 6.4.1).
> This aspect is a future-proofing against having to consider if future code
> changes need to consider __all__ or not. With the empty __init__.py there
> are no potential forcing issues with respect to future code updates.
>
> Well, you don't import anything, then there is no reason to define __all__
> in the __init__ file. In that case the file should just be empty. As the
> paragraphs that you pasted above explain, the purpose of __all__ is to
> limit what gets imported when I do "from x3d import *" for example, if the
> following is my __init__.py:
>
> from math import sqrt
>
> from
>
> __all__ = ['answer', 'MyClass']
>
> answer = 5
>
> class MyClass():
>
> def calc(self, x):
>
> return sqrt(x)
>
>
>
>
>
> then "from xxx import *" gives me "answer" and "MyClass". Without the
> __all__ I would get "answer" "MyClass" and
> "sqrt" i.e. everything that's available inside __init__.py. Often, that's
> not what we want because sqrt is nothing specific from our package. So
> __all__ restricts that. In the version of x3d.py that I have the __init__
> files lists a whole bunch of things in the __all__, but does not import
> them. So why is there an __all__ at all?
>
> However, the most common case is to selectively import things into
> __init__, often including things like a version string (as __version__) and
> the most important classes or functions that user may want in the flat
> namespace.
>
> Examples from pretty standard Python packages:
>
> https://github.com/urllib3/urllib3/blob/main/src/urllib3/__init__.py#L27
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Furllib3%2Furllib3%2Fblob%2Fmain%2Fsrc%2Furllib3%2F__init__.py%23L27&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=uYhnkOVFkxgmZKLAUWSU4tm53XShv8hNyxM03yU9T0Q%3D&reserved=0>
>
> https://github.com/pypa/setuptools/blob/main/setuptools/__init__.py#L24
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpypa%2Fsetuptools%2Fblob%2Fmain%2Fsetuptools%2F__init__.py%23L24&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=z2l%2BNf9e7f%2BgyoaHvlT6YdjsLncMG87JLF3RywAcJ4I%3D&reserved=0>
>
> https://github.com/numpy/numpy/blob/main/numpy/__init__.py
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnumpy%2Fnumpy%2Fblob%2Fmain%2Fnumpy%2F__init__.py&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=wc6xFZFMtDhQ7Heu49z0PJ1HW4qabS4%2FHZgleLzBOS0%3D&reserved=0>
>
>
>
>
>
> Additionally, and as indicated by the bulk of the next two paragraphs,
> probably more importantly: I have been burned very badly in the past,
> entering Imports Hell, based on using an ill-timed from xyz import * and
> had to explore the arcane and totally insane practice of putting import
> statements at the bottom of a file to partially attempt to compensate for
> import loading circular dependencies. Putting from x3d import * into
> the __init__.py file opens the door to that netherworld which awaits
> someone unluckily coding something that creates a circular dependency. I
> can't guarantee that there is zero chance of that in coding up a solution
> for Numpy types, or other kinds of types that may be necessary in the
> future. We would hate to inadvertently force a package user into that
> special hell.
>
> I'm curious to learn how that works, since I've never seen a case where
> any other module imported anything from __init__.py Circular imports can
> happen and have happened to me, but never related to __init__, so I'm a
> little surprised. about that.
>
>
>
> - Python usually makes heavy use of duck-typing (you can use any object as
> long as it behaves as you expect) instead of explicit "isinstance" checks.
> A lot of numerical computations in Python are done using the numpy (
> https://numpy.org
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnumpy.org%2F&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=cWIOWCGX7Av%2BqnMXgyWNB0ws8djZip3eDzEaP2I4Lzo%3D&reserved=0>)
> library, which implements array-based math and is optimized in C. Numpy has
> datatypes that are essentially similar to Python int but fail an isinstance
> checks, and I can pass those in some, but not all places in x3d.py. Here is
> an example that fails:
>
> We agree that this use-case needs addressing to provide a Numpy compliant
> x3d.py experience. Please provide some feedback on following proposal,
> design goals/constraints first.
>
>
>
> 1) Yes, in the implementation code there is a point where duck-typing must
> become enforced typing, because the x3d module is for implementing a
> specified standard which has specified types.
>
> But the specific standard is not the Python type. As I understand it the
> standard says "integer" not "64 bit integer in big-endian". If I'm correct,
> any int will do (32 bin on a 32 bit machine, 64 bit on a 64 bit machine
> etc.). If that's case, then there is in fact no reason at all to case e.g.
> a numpy int64 to a Python int - on a 64-bit machine they are identical
> anyway. When I output the x3d to an xml file or a json file, they will all
> be converted to a string anyway. Then, it's even less important what type
> of int they where initially, because you won't even see the difference
> between an int64 and an int128.
>
> So the type checking problem is unavoidable, it just comes down to at what
> layer of the code does a particular type run into an unavoidable
> requirement to finally do the type-check for a verified
> per-standard-construction. The requirement has been addressed in code for
> the transition from basic types to the specified X3D types. We know that
> so far this code works, so it would be nice to not rewrite all, or even
> any, of that working code.
>
>
>
> 2) Users who are using native Python types as inputs should not feel any
> performance change when adding the necessary Numpy type use functionality
> to module x3d. Adding mandatory casts, or even conditionals for deciding
> casts, could add significant overhead if millions+ of values are processed
> so targeting only where the price must be paid no-matter-what is advisable.
>
> Performance is always and important consideration. However, it seems to me
> that this is premature optimization. We are not the first people thinking
> about this and it's not a bottleneck. On my machine, "if isinstance(a,
> int)" is about 1 ns faster than "int(a) = a" if a is an int already (that's
> the case of possible performance loss, if a is not an int already, then the
> current implementation fails anyway).
>
> %timeit if isinstance(a, int): pass
>
> 50.3 ns ± 0.266 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>
> %timeit if int(a) == a: pass
>
> 51.4 ns ± 0.172 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
>
>
>
> That 1 ns is only 2% of the total runtime of the if statement and < 0.04 %
> of the total runtime of executing something useful where that statement
> matters. Here, I making the smallest IndexedFaceSet I can. If that takes 1
> ns longer, that's OK. If I can afford to run a 26.3 microsecond operation a
> few million times, I can afford to run a 26.301 microsecond operation - the
> difference is within the noise of the measurement.
>
> %timeit out = x3d.IndexedFaceSet(coord=x3d.Coordinate(point=[(0, 0, 0),
> (0, 1, 0), (1, 1, 0)]), coordIndex=[0, 1, 2])
>
> 26.3 µs ± 185 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
>
>
>
>
>
> 3) There must be some other communities out there, or in the future, that
> will have similar type issues specific to their use-case implementation,
> our Numpy solution should show the way forward for those, and in
> implementing those future updates require as little touching of
> verified-correct code as possible.
>
> Numpy by far the most important one. While counting the importance of
> libraries is a tricky business, in e.g. PEP 465
> https://www.python.org/dev/peps/pep-0465/#but-isn-t-matrix-multiplication-a-pretty-niche-requirement
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python.org%2Fdev%2Fpeps%2Fpep-0465%2F%23but-isn-t-matrix-multiplication-a-pretty-niche-requirement&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=tbqtlpSosjx1rfXISqG4tSnqkNaM%2Bp4fYpx%2Fh0qfPO8%3D&reserved=0>
> the authors found that numpy was the forth most used library - ranking far
> before most packages of even the standard library (only sys, os, and re and
> imported more often). That does not mean that there is not and won't ever
> be other ways to do numerics in Python ever, but it does mean the numpy is
> a pretty save bet.
>
>
>
> *Proposal*: Adding a submodule in the x3d module which serves as the
> interface layer between Numpy users and x3d.py. Numpy users should only
> need to import their 'shim' from the x3d module (As a non-binding name
> example: import x3d.numpy2x3d as x3d_ I don't think I like the specific
> name either, but we all get the idea) and it will provide Numpy-compliant
> type input, do any necessary type casting on parameters, and then call
> working x3d.py code. It should have an equivalent call for every public API
> function/method in x3d.py, but only include the minimal necessary
> type-casting code to make the call. Once developed, well tested and
> verified, the long term maintenance tail of this shim file should be mostly
> restricted to times when Numpy makes changes to its type system (VERY
> rare), and if the x3d.py API gets any public additions (bugs are bugs, and
> that's not the tail I'm addressing here as we have the duty to kill as many
> as possible early.)
>
> I think that will work - but it's not the most convenient for the user. I
> think the most convenient would be to iterate over any input, and don't
> check that the input is a list
>
> This works:
>
> x3d.Coordinate(point=[[0, 0, 0], (0, 1, 0), (1, 1, 0)])
>
> This fails:
>
> x3d.Coordinate(point=([0, 0, 0], (0, 1, 0), (1, 1, 0)))
>
> X3DTypeError: ([0, 0, 0], (0, 1, 0), (1, 1, 0)), type=<class 'tuple'> is not a valid Python list for MFVec3f
>
>
>
> Why not allow a tuple instead of a list here? The inner coordinates can be either tuples or lists.
>
>
>
> I don't understand what that typechecking is good for here.
>
>
>
> Moritz
>
>
>
>
>
> An added benefit of this flavor of solution is that your participation is
> not within the autogenerated x3d.py layer, so no concerns of breaking a
> working x3d.py file. Writing this to be autogenerated as well is probably
> best practice, that way it significantly reduces eyeball-missed public API
> functions. Don is the expert in the auto-generation aspect. I helped him
> with x3d.py by describing the repeatable stylistic issues, conventions and
> some advantageous techniques to provide low-cost syntactic guarantees
> within chunks of code, he made the magic auto-happen, we debugged pairs
> style with him driving. Setting up the Numpy-isms is likely a very similar
> flavor of project, but more favorably scoped as start-type and
> destination-type are both going to be defined the majority of the time,
> your expertise will be quite valuable in this process.
>
> I have not started coding, but I would be surprised if more than 100 line
> of code are needed for this.
>
>
>
> Looking forward to seeing your thoughts. Have a great weekend!
>
>
>
> v/r Loren
>
>
>
>
>
> *From: *Hans Moritz Guenther <hgunther at mit.edu> <hgunther at mit.edu>
> *Date: *Monday, February 7, 2022 at 7:06 AM
> *To: *Don Brutzman <brutzman at nps.edu> <brutzman at nps.edu>, "Peitso, Loren
> (CIV)" <lepeitso at nps.edu> <lepeitso at nps.edu>
> *Subject: *x3d.py package: Some feedback and suggestion for improvement
>
>
>
> NPS WARNING: *external sender* verify before acting.
>
>
>
> Hi,
>
> I'm starting to use the x3d.py library to generate x3d output.
>
> For a little bit of background, I'm an astronomer and one of the things I
> do is ray-tracing of the designs for new instruments. It is then very
> useful to have some form of output that allows me to put images of how the
> instrument looks and what path the photons take into presentations or on
> the web. As it turns out, X3D is a great format for that, see e.g.
> https://space.mit.edu/home/guenther/ARCUS/3Dview.html
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspace.mit.edu%2Fhome%2Fguenther%2FARCUS%2F3Dview.html&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=1oRxV8StuVpt2%2Bcblf17BDtdtzWxT8xwAJViNaZsG2A%3D&reserved=0>
>
> In that past, I've generated the X3D output through a python package
> called mayavi (https://docs.enthought.com/mayavi/mayavi/
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.enthought.com%2Fmayavi%2Fmayavi%2F&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=XLLgQdaHYlWPWGP9vTf3Hg4WLKa%2Fz01xgBGVbOsSSBE%3D&reserved=0>)
> but that is a rather heavy dependency for this very limited purpose. So,
> I'm trying to convert to use your x3d.py library instead.
>
> I'm very much a Python programmer using what people call the "scientific
> stack" in Python (the libraries numpy, scipy, pandas, etc.) with very
> little experience in 3D visualization or web-programming. First, let me
> thank you for your effort to make x3d.py at all. In general it works great!
>
> I'd like to mention a few glitches I've run into so far, and since I did
> not see in your package description what's the preferred way to give
> feedback I decided to just email you. Please let me know if you prefer me
> to raise those in a different forum (mailing list, sourceforge ticket ...).
>
> I might be able to help with some of this, but since you say your package
> is autogenerated, there is little use for me to change the code I have;
> presumably all changes need to be done to the code that you use to
> autogenerate things.
>
> OK, here are a few things I noticed:
>
> - the __init__.py file. The file is present, but not correct. It lists all
> the classes in __all__ but does not actually import them. Thus, that can't
> actually be used.
>
> In [1]: import x3d
>
> In [2]: x3d.IndexedTriangleSet
> ---------------------------------------------------------------------------
> AttributeError Traceback (most recent call last)
> <ipython-input-2-93a98d5f0414> in <module>
> ----> 1 x3d.IndexedTriangleSet
>
> Instead, I need to use "x3d.x3d". That's of course possible but a little
> cumbersome.
>
> In [4]: import x3d.x3d
> x3d.py package loaded, have fun with X3D Graphics!
>
> In [5]: x3d.x3d.IndexedTriangleSet
> Out[5]: x3d.x3d.IndexedTriangleSet
>
> I think all that's missing is the following line in the __init__.py
>
> from x3d import *
>
> which will import everything that's defined in x3d.py into the __init__.py
> namespace and than the user an get it from "import x3d" which gives you the
> names defined in x3d/__init__.py
>
> - IndexTriangleSet: Coordinates are given as a list of tuples [(x1, y1,
> z1), (x2, y2, y3), ...]. I think that makes sense. It orders stuff in a
> natural way and is easy to understand. However, the index needs to be given
> as a flat list [ind11, ind12, ind13, ind21, ind22, ind23, ...]. Since there
> are three indices per triangle, I think the same list of tuples as for the
> coordinates would make more sense [( ind11, ind12, ind13), (ind21, ind22,
> ind23) ...]
>
> - Python usually makes heavy use of duck-typing (you can use any object as
> long as it behaves as you expect) instead of explicit "isinstance" checks.
> A lot of numerical computations in Python are done using the numpy (
> https://numpy.org
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnumpy.org%2F&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=cWIOWCGX7Av%2BqnMXgyWNB0ws8djZip3eDzEaP2I4Lzo%3D&reserved=0>)
> library, which implements array-based math and is optimized in C. Numpy has
> datatypes that are essentially similar to Python int but fail an isinstance
> checks, and I can pass those in some, but not all places in x3d.py. Here is
> an example that fails:
>
>
>
> ~/mambaforge/envs/kitchensink/lib/python3.10/site-packages/x3d/x3d.py in __init__(self, ccw, colorPerVertex, index, normalPerVertex, solid, color, coord, fogCoord, normal, texCoord, attrib, DEF, USE, IS, metadata, class_, id_, style_)
>
> 48191 self.ccw = ccw
>
> 48192 self.colorPerVertex = colorPerVertex
>
> > 48193 self.index = index
>
> 48194 self.normalPerVertex = normalPerVertex
>
> 48195 self.solid = solid
>
>
>
> ~/mambaforge/envs/kitchensink/lib/python3.10/site-packages/x3d/x3d.py in index(self, index)
>
> 48230 if index is None:
>
> 48231 index = MFInt32.DEFAULT_VALUE()
>
> > 48232 assertValidMFInt32(index)
>
> 48233 assertNonNegative('index', index)
>
> 48234 self.__index = index
>
>
>
> ~/mambaforge/envs/kitchensink/lib/python3.10/site-packages/x3d/x3d.py in assertValidMFInt32(value)
>
> 2622 if not isinstance(each, int):
>
> 2623 # print(flush=True)
>
> -> 2624 raise X3DTypeError('MFInt32 list has contained value=' + str(each) + ' with type=' + str(type(each)) + ' which is not a valid int')
>
> 2625 if not isValidMFInt32(value):
>
> 2626 # print(flush=True)
>
>
>
> X3DTypeError: MFInt32 list has contained value=1 with type=<class 'numpy.int64'> which is not a valid int
>
>
>
>
>
> There is an easy workaround:
>
>
>
> x3d.IndexedTriangleSet(index = [int(x) for x in myindex]
>
>
>
> but it would be nice if IndexedTriangleSet accepted my numbers directly.
>
> There are several ways of doing that, but the easiest is probably to change line 2622 to:
>
>
>
> if not int(x) == x:
>
>
>
> That will work for any object that can be converted to an int, including
> numpy, python decimal, fraction, ..
>
> - Numpy: One might consider taking numpy arrays directly, i.e. instead of
>
> x3d.Coordinate(point=[(x1, y1, z1), (x2, y2, y3), ...])
>
> one could do
>
> x3d.Coordinate(point=arr)
>
> where arr is a (3, n) numpy array. Now, if done naively that would require
> numpy as a dependency to x3d.py and it's probably good to avoid that.
> However, there are ways to accept numpy arrays without requiring numpy.
> That's a little more involved, but can be done (for example using
> decorators or a separate module (x3d.numpy_interface) or separate package
> (x3d-numpy)). Not sure if it's worth the effort at this point - that
> depends on what your future plans for this package are and how fast this is
> developing.
>
> - Changelog. From the pypi entry and the docs on
> https://www.web3d.org/x3d/stylesheets/python/python.html
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.web3d.org%2Fx3d%2Fstylesheets%2Fpython%2Fpython.html&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=KOCzTqTvVtFHM1bfaAU3W3LLWX2O7Ml9s%2F2TNUa4KFc%3D&reserved=0>
> it was not quite clear to me how stable the package is or where I would see
> changes listed, for example, if you do the change that I suggested above
> (accepting index values as tuples instead of a flat list) that would break
> the interface. Would you do that? Where would I find a list of changes from
> one version to another?
>
> - Jupyter notebook: I'm working on a wrapper to automatically display any
> x3d representation in the jupyter notebook. It's only a few lines of code,
> and I'm happy to share (it could be integrated into the Python package x3d
> easily), but I want to use it a little longer for myself to make sure I
> have the kinks worked out so I don't pass code with major bugs to you.
>
> Please let me know if there is anything I can do to help with this awesome
> package, that really makes generating X3D from Python so much simpler
> already.
>
> Yours,
>
> Moritz
>
>
>
> --
>
> Hans Moritz Günther
>
> Massachusetts Institute of Technology
>
> Kavli Institute for Astrophysics and Space Research
>
> 77 Massachusetts Avenue <https://www.google.com/maps/search/77+Massachusetts+Avenue?entry=gmail&source=g>
>
> NE83-569
>
> Cambridge, MA 02139
>
> hgunther at mit.edu
>
> https://space.mit.edu/home/guenther/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspace.mit.edu%2Fhome%2Fguenther%2F&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=8bq%2BK5n3GMrYU5PEQdt5BLqX58XTKcXTkcGybZfUxG0%3D&reserved=0>
>
> --
>
> Hans Moritz Günther
>
> Massachusetts Institute of Technology
>
> Kavli Institute for Astrophysics and Space Research
>
> 77 Massachusetts Avenue <https://www.google.com/maps/search/77+Massachusetts+Avenue?entry=gmail&source=g>
>
> NE83-569
>
> Cambridge, MA 02139
>
> hgunther at mit.edu
>
> https://space.mit.edu/home/guenther/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspace.mit.edu%2Fhome%2Fguenther%2F&data=04%7C01%7Cbrutzman%40nps.edu%7Cdb8f437f034c41d651cb08d9edda131d%7C6d936231a51740ea9199f7578963378e%7C0%7C0%7C637802345006585381%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=8bq%2BK5n3GMrYU5PEQdt5BLqX58XTKcXTkcGybZfUxG0%3D&reserved=0>
>
> _______________________________________________
> x3d-public mailing list
> x3d-public at web3d.org
> http://web3d.org/mailman/listinfo/x3d-public_web3d.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://web3d.org/pipermail/semantics-public_web3d.org/attachments/20220223/b4bac082/attachment-0001.html>
More information about the semantics-public
mailing list