Automated Acquisition and Real-time Rendering of Spatially Varying Optical Material Behavior
ACM SIGGRAPH 2018 Posters
International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) <45, 2018, Vancouver, BC, Canada>
We created a fully automatic system for acquisition of spatially varying optical material behavior of real object surfaces under a hemisphere of individual incident light directions. The resulting measured material model is flexibly applicable to arbitrary 3D model geometries, can be photorealistically rendered and interacted with in real-time and is not constrained to isotropic materials.
CultArc3D_mini: Fully Automatic Zero-Button 3D Replicator
3D scanning and 3D printing are two rapidly evolving domains, both generating results with a huge and growing spectrumof applications. Especially in Cultural Heritage, a massive and increasing amount of objects awaits digitization for variouspurposes, one of them being replication. Yet, current approaches to optical 3D digitization are semi-automatic at best andrequire great user effort whenever high quality is desired. With our solution we provide the missing link between both domains,and present a fully automatic 3D object replicator which does not require user interaction. The system consists of ourphotogrammetric 3D scanner CultArc3D_mini that captures an optimal image set for 3D geometry and texture reconstructionand even optical material properties of objects in only minutes, a conveyor system for automatic object feed-in and -out,a 3D printer, and our sensor-based process flow software that handles every single process step of the complex sequencefrom image acquisition, sensor-based object transportation, 3D reconstruction involving different kinds of calibrations, to3D printing of the resulting virtual replica immediately after 3D reconstruction. Typically, one-button machines require theuser to start the process by interacting over a user interface. Since positioning and pickup of objects is automatically registered,the only thing left for the user to do is placing an object at the entry and retrieving it from the exit after scanning. Shortly after,the 3D replica can be picked up from the 3D printer. Technically, we created a zero-button 3D replicator that provides highthroughput digitization in 3D, requiring only minutes per object, and it is publicly showcased in action at 3IT Berlin.
Example-based Synthesis of Seamless Texture Variations and Application to the Acquisition of Optical Material-Properties
Darmstadt, TU, Master Thesis, 2018
This work extends an existing workflow for acquisition, synthesis and rendering of Approximate Bi-directional Texturing Functions (ABTF), which represent a lower-dimensional alternative to Bi-directional Texturing Functions (BTF). In the first steps, image corrections and registration are presented to optimize the current setup. Texture synthesis needs to consider the surface geometry and reflection parameters to generate consistent images for all illumination directions. Furthermore, a method for generating seamless texture tiles and randomly spread them on a target surface to create renderings with less noticeable repetitions in texture is discussed in detail. To handle the enormous memory consumption of multiple ABTF datasets a fitting scheme is proposed specifically designed to reconstruct the most prominent effects captured in the ABTF model.
Extension of the ABTF Material Acquisition and Rendering Process to CultArc3D Image Data
Darmstadt, Hochschule, Bachelor Thesis, 2018
The importance of photorealistic 3D rendering of different materials is increasing, as there are various domains of application, such as the textile and the 3D games industry. In order to be able to do real-time rendering involving a physical material, a method for its acquisition and realistic rendering on 3D geometry is required. So far a single-camera system called ABTF Scanner is already able to acquire flat materials that are anisotropic (appearance dependent on rotation around surface normal) using a turntable, which makes it possible to map the acquired material onto an arbitrary 3D geometry during real-time rendering. Another scanner system consisting of multiple cameras called CultArc3D can also be used for this purpose. Due to its structure, allowing for lighting over a hemisphere by turning an arc equipped with light sources, there is no need for a turntable to acquire materials, as opposed to the single-camera system that achieves hemispheric illumination by combining a fixed quarter light arc with a rotary. In order to make images acquired by CultArc3D usable for real-time rendering, this thesis extends the software implemented for the ABTF Scanner. The extension was done in a way that makes the software multi functional in that it is now able to do real-time rendering for materials acquired by either of the above mentioned scanner system. For the first time images taken by CultArc3D can be used for renderings of ABTF material samples that capture material behavior for a comparable set of virtual light directions. Additionally a new shader (computer program used for 3D rendering) is implemented to provide real-time rendering with respect to the different data structure imposed by the concept of CultArc3D. The experimental evaluation shows that real-time rendering with the images acquired by CultArc3D can lead to better results compared to images taken by the ABTF Scanner, because the back-rotation of images, introduced by the rotary in the ABTF Scanner setup, is not required by CultArc3D. Thus, a number of calibrations and alignment steps that possibly introduce visual artifacts if not performed correctly, can be avoided. As a result CultArc3D can now be used for ABTF real-time rendering in addition to its capability of acquiring geometry, texture and a number of different optical material models. The software can be extended for different viewing perspectives during rendering in future work, due a hemispherical distribution of camera perspectives around the object.
Synthesis and Rendering of Seamless and Non-Repetitive 4D Texture Variations for Measured Optical Material Properties
SIGGRAPH Asia 2018 Technical Briefs
We have lifted the one weakness of an existing fully automatic acquisition system for spatially varying optical material behavior of real object surfaces. While its expression of spatially varying material behavior with spherical dependence on incoming light as 4D texture (ABTF material model) allows flexible mapping on arbitrary 3D geometries, photo-realistic rendering and interaction in real-time, this very method of texture-like representation exposed it to common problems of texturing, striking in two levels. First, non-seamless textures create visible border artifacts. Second, even a perfectly seamless texture causes repetition artifacts due to side-by-side distribution in large numbers over the 3D surface. We solved both problems through our novel texture synthesis that generates a set of seamless texture variations randomly distributed on the surface at shading time. When compared to regular 2D textures, the inter-dimensional coherence of the 4D ABTF material model poses entirely new challenges to texture synthesis, which includes maintaining the consistency of material behavior throughout the space spanned by the spatial image domain and the angular illumination hemisphere. In addition, we tackle the increased memory consumption caused by the numerous variations through a fitting scheme specifically designed to reconstruct the most prominent effects captured in the material model.
3D Mass Digitization: A Milestone for Archeological Documentation
VAR. Virtual Archaeology Review [online]
In the heritage field the demand for fast and efficient 3D digitization technologies for historic remains is increasing. Besides, 3D digitization has proved to be a promising approach to enable precise reconstructions of objects. Yet, unlike the digital acquisition of cultural goods in 2D widely used today, 3D digitization often still requires a significant investment of time and money. To make it more widely available to heritage institutions, the Competence Center for Cultural Heritage Digitization at the Fraunhofer Institute for Computer Graphics Research IGD has developed CultLab3D, the world's first fully automatic 3D mass digitization facility for collections of three-dimensional objects. CultLab3D is specifically designed to automate the entire 3D digitization process thus allowing users to scan and archive objects on a large-scale. Moreover, scanning and lighting technologies are combined to capture the exact geometry, texture, and optical material properties of artefacts to produce highly accurate photo-realistic representations. The unique setup allows shortening the time needed for digitization to several minutes per artefact instead of hours, as required by conventional 3D scanning methods.
Acceleration of 3D Mass Digitization Processes: Recent Advances and Challenges
Mixed Reality and Gamification for Cultural Heritage
In the heritage field, the demand for fast and efficient 3D digitization technologies for historic remains is increasing. Besides, 3D has proven to be a promising approach to enable precise reconstructions of cultural heritage objects. Even though 3D technologies and postprocessing tools are widespread and approaches to semantic enrichment and Storage of 3D models are just emerging, only few approaches enable mass capture and computation of 3D virtual models from zoological and archeological findings. To illustrate how future 3D mass digitization systems may look like, we introduce CultLab3D, a recent approach to 3D mass digitization, annotation, and archival storage by the Competence Center for Cultural Heritage Digitization at the Fraunhofer Institute for Computer Graphics Research IGD. CultLab3D can be regarded as one of the first feasible approaches worldwide to enable fast, efficient, and cost-effective 3D digitization. lt specifically designed to automate the entire process and thus allows to scan and archive large amounts of heritage objects for documentation and preservation in the best possible quality, taking advantage of integrated 30 visualization and annotation within regular Web browsers using technologies such as WebGI and X3D.
c-Space: Time-evolving 3D Models (4D) from Heterogeneous Distributed Video Sources
Eurographics Workshop on Graphics and Cultural Heritage (GCH) <14, 2016, Genova, Italy>
We introduce c-Space, an approach to automated 4D reconstruction of dynamic real world scenes, represented as time-evolving 3D geometry streams, available to everyone. Our novel technique solves the problem of fusing all sources, asynchronously captured from multiple heterogeneous mobile devices around a dynamic scene at a real word location. To this end all captured input is broken down into a massive unordered frame set, sorting the frames along a common time axis, and finally discretizing the ordered frame set into a time-sequence of frame subsets, each subject to photogrammetric 3D reconstruction. The result is a time line of 3D models, each representing a snapshot of the scene evolution in 3D at a specific point in time. Just like a movie is a concatenation of time-discrete frames, representing the evolution of a scene in 2D, the 4D frames reconstructed by c-Space line up to form the captured and dynamically changing 3D geometry of an event over time, thus enabling the user to interact with it in the very same way as with a static 3D model. We do image analysis to automatically maximize the quality of results in the presence of challenging, heterogeneous and asynchronous input sources exhibiting a wide quality spectrum. In addition we show how this technique can be integrated as a 4D reconstruction web service module, available to mobile end-users.
CultLab3D - On the Verge of 3D Mass Digitization
Eurographics Symposium on Graphics and Cultural Heritage (GCH) <12, 2014, Darmstadt, Germany>
Acquisition of 3D geometry, texture and optical material properties of real objects still consumes a considerable amount of time, and forces humans to dedicate their full attention to this process. We propose CultLab3D, an automatic modular 3D digitization pipeline, aiming for efficient mass digitization of 3D geometry, texture, and optical material properties. CultLab3D requires minimal human intervention and reduces processing time to a fraction of today's efforts for manual digitization. The final step in our digitization workflow involves the integration of the digital object into enduring 3D Cultural Heritage Collections together with the available semantic information related to the object. In addition, a software tool facilitates virtual, location-independent analysis and publication of the virtual surrogates of the objects, and encourages collaboration between scientists all around the world. The pipeline is designed in a modular fashion and allows for further extensions to incorporate newer technologies. For instance, by switching scanning heads, it is possible to acquire coarser or more refined 3D geometry.
Synthesis of Periodic Textures Using Image Decomposition and Neighborhood Synthesis
Darmstadt, TU, Master Thesis, 2013
This thesis deals with the automatic generation of texture tiles based on a given two-dimensional texture sample. By simply stitching the tiles together, arbitrarily sized planes can be tessellated in little time at low storage requirement. To ensure satisfying results, several demands have to be considered: Most essential for a successful tiling is a good boundary handling to prevent visible seams. Among color adjustment, it is a special quest to maintain features beyond the borders. Moreover, the tile should contain enough information about the texture's topology in order to be able to realistically reconstruct it. Since there is a variety of textures with different properties and demands, two approaches are proposed that together cover the whole spectrum: The first one initially corrects the input image regarding rotation and warping and then decomposes it in a periodic part and a smooth residual, where the actual decomposition depends on several parameters. The periodic part can then be used as a tile. The second approach reconstructs a transition between opposing borders by extracting matching patches from the input image and stitching them along the borders. Since unwanted repetitions might occur when simply stitching the tiles together, an additional re-synthesis step is proposed at the end of this work that re-synthesizes the inner part of a tile in different ways and thus generates a stack where tiles can be randomly chosen from and stitched together until the desired space is filled.
High Resolution Acquisition of Detailed Surfaces with Lens-Shifted Structured Light
Computers & Graphics
We present a novel 3D geometry acquisition technique at high resolution based on structured light reconstruction with a low-cost projector-camera system. Using a 1D mechanical lens-shifter extension in the projector light path, the projected pattern is shifted in subpixel scale steps with a granularity of up to 2048 steps per projected pixel, which opens up novel possibilities in depth accuracy and smoothness for the acquired geometry. Combining the mechanical lens-shifter extension with a multiple phase shifting technique yields a measuring range of 120×80 mm while at the same time providing a high depth resolution of better than 100µm. Reaching beyond depth resolutions achieved by conventional structured light scanning approaches with projector-camera systems, depth layering effects inherent to conventional techniques are fully avoided. Relying on low-cost consumer products only, we reach an area resolution of down to 55µm (limited by the camera). We see two main benefits. First, our acquisition setup can reconstruct finest details of small cultural heritage objects such as antique coins and thus digitally preserve them in appropriate precision. Second, our accurate height fields are a viable input to physically based rendering in combination with measured material BRDFs to reproduce compelling spatially varying, material-specific effects.
Interactive Semantic Enrichment of 3D Cultural Heritage Collections
International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST) <13, 2012, Brighton, UK>
Virtual Surrogates of Cultural Heritage (CH) objects are seriously being considered in professional activities such as conservation and preservation, exhibition planning, packing, and scholarly research, among many other activities. Although this is a very positive development, a bare 3D digital representation is insufficient and poor for fulfilling the full range of professional activities. In this paper, we present the first interactive semantic enrichment tool for 3D CH collections that is fully based on the CIDOC-CRM schema and that fully supports its sophisticated annotation model. The tool eases the user interaction, allowing inexperienced users without previous knowledge on semantic models or 3D modeling to employ it and to conceive it for the professional workflow on 3D annotations. We illustrate the capabilities of our tool in the context of the Saalburg fort, during Roman times (2nd century AD), for the protection of the Limes on the Taunus hills in Germany.
Iterative Scan Planning for Automated 3D Reconstruction
Dijon, Univ. de Bourgogne, Master Thesis, 2012
3D Scanning is one of the most important and interesting areas in modern Computer Vision research. 3D Scanning mainly deals with creating a virtual CAD/Polygonal mesh model of real object. For proper preservation and restoration of cultural objects, it is very important to have detailed information about the object in 3D. The current methods available to scan and represent the digital 3D model of such objects are very cumbersome and expensive. In this thesis, we present a method for Iterative scan planning for Automated 3D reconstruction of Cultural Heritage Objects. The main area we have focussed is automated 3D scanning. Primary goal is to implement automated scanning with minimum human intervention in the entire process. The method intended will be similar to the concept of Assembly line in production, where the automation and separation of individual tasks will help to speed up the process of 3D scanning. We have implemented a method similar to mass vector chain approach for iterative scanning. Our algorithm predicts evolution of 3D surface based on previously scanned surfaces. This idea coupled with several approaches from graph theory provides robust estimation for the surface evolution, which can be used in determining next pose. Another part of the thesis focuses on "Geometric lossy compression of 3D meshes using convex hulls". This method is entirely new proposed by the author. This method is based on use of ellipsoidal convex hulls to get lossy compression of the given geometric 3D mesh. The compressed version of mesh can later be used for multi-level or multi-resolution viewing and easy transportation over web. The use of ellipsoidal convex hulls combined with the method of ray-casting provides at least 35 percent compressions for the same number of vertices, although the original vertices are not retained. Instead they will be replaced by the new vertices obtained by the method of ray-casting onto the surface of given 3D mesh.
Removing the Example from Example-based Photometric Stereo
Trends and Topics in Computer Vision
Reconstruction and Modeling of Large-Scale 3D Virtual Environments (RMLE) <2010, Heraklion, Greece>
We introduce an example-based photometric stereo approach that does not require explicit reference objects. Instead, we use a robust multi-view stereo technique to create a partial reconstruction of the scene which serves as sceneintrinsic reference geometry. Similar to the standard approach, we then transfer normals from reconstructed to unreconstructed regions based on robust photometric matching. In contrast to traditional reference objects, the scene-intrinsic reference geometry is neither noise free nor does it necessarily contain all possible normal directions for given materials.We therefore propose several modifications that allow us to reconstruct high quality normal maps. During integration, we combine both normal and positional information yielding high quality reconstructions. We show results on several datasets including an example based on data solely collected from the Internet.
Dense 3D Reconstruction and Object Recognition using a Minimum Set of Inside-Out Images
Pilani, BITS Univ., Bachelor Thesis, 2011
Dense 3D Reconstruction of environments is important for various applications like augmented reality, artefact digitization and object classification. Object classification in particular allows for scene understanding. This work proposes the development of a pipeline for image based 3D reconstruction and object recognition. The 2D images under consideration are the inside out images of the interior of a room. A dense 3D reconstruction allows the description of the room as point clouds on which the object recognition algorithms are implemented. To allow for flexibility in terms of image acquisition methods, the algorithm is robust to the type of image input as well as the number of images. Matching algorithms like Scale Invariant Feature Transform (SIFT) provide accurate correspondences while Structure from Motion (SFM) algorithms use these correspondences to estimate precise camera pose and Multi-view Stereo (MVS) methods take images with pose as input and produce dense 3D models. The reconstructed scenes are then acted upon by the 3D feature extractor and the features are compared with pre-trained classifiers from a database to carry out object recognition. The pipeline has been developed to allow for different types of Multi-view Stereo input images. While plane images allow for cheap equipment, spherical and cylindrical panoramic images allow for ease in image acquisition. We have used a modified version of the existent Open Street Map (OSM) Bundler for Structure from Motion and an overlapping view clustering problem for the Multi-view Stereo. Iterative Closest Point algorithm allows for integrating the depth maps to generate the mesh model. The pipeline also allows for an input generated using the Microsoft Xbox Kinect. We have used the Cluster Viewpoint Feature Histogram algorithm (CVFH) for the object recognition and also proposed the use of Normal Aligned Radial Features (NARF). We also study the prospect of using 2D to 3D feature correlation to find objects in the 3D generated model of the room from a 2D image of that room. This work also shows the results of a comparative study undertaken between the different possible methods to complete the task. We also study the different image geometries to further explore the invariance to camera models. Finally the pipeline has been integrated in the Rapid Prototyping Environment (RPE) framework of the Department Interactive Engineering Technologies as a plug-in to provide additional functionality.
A Full HDR Pipeline from Acquisition to Projection
Siggraph 2010. Full Conference DVD-ROM
International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) <37, 2010, Los Angeles, CA, USA>
In this publication we present one of the first full HDR visualization systems starting with HDR material and light acquisition, providing a HDR light simulation and rendering pipeline and finally displaying maximum fidelity image quality with color gamut enhanced HDR projection technology to bring the total dynamic range to over 5.000.000:1. We demonstrate these capabilities in the fields of car design and architecture.
High Resolution Acquisition of Detailed Surfaces with Lens-Shifted Structured Light
International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST) <11, 2010, Paris, France>
We present a novel 3D geometry acquisition technique at high resolution based on structured light reconstruction with a low-cost projector-camera system. Using a 1D mechanical lens-shifter extension in the projector light path, the projected pattern is shifted in fine steps at sub-pixel scale with a granularity of down to 2048 steps per projected pixel, which opens up novel possibilities in depth accuracy and smoothness for the acquired geometry. Combining the mechanical lens-shifter extension with a multiple phase shifting technique yields a measuring range of 120x80 mm while at the same time providing a high depth resolution of better than 100 micron. Reaching far beyond depth resolutions achieved by conventional structured light scanning approaches with projector-camera systems, depth layering effects inherent to conventional techniques are fully avoided. Relying on low-cost consumer products only, we reach an area resolution of down to 55 micron (limited by the camera). We see two fields of benefit. Firstly, our acquisition setup can reconstruct finest details of small Cultural Heritage objects such as antique coins and thus digitally preserve them in appropriate precision. Secondly, our accurate height fields can be viable input to physically based rendering in combination with measured material BRDFs to reproduce compelling spatially varying, material-specific effects.
Scene Reconstruction from Community Photo Collections
The literally billions of images available from online photo-sharing sites offer an unprecedented wealth of information but also add additional layers of complexity for reconstruction applications.
Surface Geometry Acquisition Using an Analog Lens-Shifting Device
Darmstadt, TU, Bachelor Thesis, 2010
Mit der Entwicklung immer leistungsfähigerer und flexiblerer Grafikhardware können zunehmend komplexere Darstellungsalgorithmen umgesetzt werden. Zusätzlich verlangen die heute weit verbreiteten hochauflösenden Displays sehr datenintensive Inhalte. Dies führt dazu, dass die Erstellung und Akquisition von Daten für eine realistische Darstellung immer mehr an Bedeutung gewinnt. Eine besonders wichtige Rolle spielt dabei die naturgetreue Modellierung und Beleuchtung von Oberflächen. Viele Bemühungen konzentrieren sich dabei auf die Vermessung und Darstellung von Reflexionseigenschaften jedoch sind auch geometrische Aspekte der Oberflächenstruktur ausschlaggebend für die Berechnung realistischer Bilder. Zwar existieren eine Reihe von Algorithmen und Systemen zur Geometrievermessung jedoch sind diese meist nicht geeignet um feine Details ausreichend aufzulösen. Deswegen beschäftigt sich diese Arbeit mit der Entwicklung und Implementierung eines Rekonstruktionsalgorithmus für einen 3D-Scanner der unter Verwendung einer speziellen Projektorerweiterung den hohen Genauigkeitsanforderungen gerecht werden kann.
A Combined Multi-View Stereo and Photometric Stereo Approach
Darmstadt, TU, Master Thesis, 2009
Two approaches to 3D-reconstruction from images, Multi-view Stereo and Photometric Stereo, are combined as a means of completing missing areas within the partly-complete reconstruction returned by the first approach. This is realized by exploiting the Photometric Stereo principle of orientation consistency, stating that image positions with similar appearance have similar surface orientation on the original object surface. Completion of reconstruction is thus achieved by finding pairs of image positions with similar appearance, one with available reconstruction serving as source, the other one with reconstruction missing as target for transfer of normals.
Satisfaction of Continuity Constraints Controlling the Transition Between Free-form Surfaces for Virtual Styling
Darmstadt, TU, Bachelor Thesis, 2006
The Bachelor thesis realized a consistency mechanism in the domain of Virtual Styling. It is able to check the feasibility of free-form creation and deformation operations considering the conformance of newly introduced continuity constraints to existing ones, and propagates deformations through the constraint graph by applying necessary surface modifications such that the continuity constraints at the transition between adjacent free-form surfaces are satisfied, leading the system into a consistent state. Some measures have been adopted to ensure a performing consistency mechanism and constraint satisfaction suitable for virtual conceptual design. One is to explore the representation of the constraint satisfaction problem in an extended constraint hyper graph. Together with the rules defined for the consistent growth and modification of the free-form model, the approach allows for an efficient consistency verification and propagation of changes throughout the constraint graph. The consistency mechanism could further be optimized by exploring the geometric properties of adjacent NURBS surfaces with continuous transition, where the affected area of shape deformations is limited to the 2-neighbourhood of a modified shape node in the constraint graph, reducing the constraint satisfaction problem to a well defined sub-graph of a given constraint graph.