AI-supported approach for human-building interaction implemented at furniture scale

Human-Building Interaction (HBI) relies on sensor-actuator networks that are increasingly supported by Artificial Intelligence (AI). This paper presents a novel AI-supported Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) approach for reconfigurable furniture. It involves a multidisciplinary approach that relies on the integration of various domains such as architecture, robotics, computer, and material science. It contributes to the advancement of HBI by employing spatial reconfiguration relying on AI and lightweight material design, which is of relevance, particularly when the furniture consists of non-identical but similar components that are re−/ configured in a variety of possible combinations.

1 Introduction

Human-Building Interaction (HBI) relying on sensor-actuator networks is increasingly supported by Artificial Intelligence (AI) at various scales (Alavi et al., 2019). In the case study presented in this paper, the novel AI-supported approach relies on Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) methods that are employed for developing reconfigurable furniture. These methods integrate robotics in building processes and buildings (Bier and Knight, 2014; Robotic Building and Bier, 2018; Pillan et al., 2020) by (a) computationally designing and robotically constructing building components (D2RP&A), and (b) by operating those components interactively (D2RO). While some of the methods and techniques used by the involved disciplines in the furniture design process are only developed conceptually, others are virtually and/ or physically prototyped.

1.1 State-of-the-art

HBI goes back to notions of Interactive Architecture (IA) relying on intelligence integrated into the built environment that has been defined already in the 70s as ‘architecture machines’ and ‘intelligent environments’ adapting to their users’ needs (Negroponte, 1975; Pask, 1975). Later, Ambient Intelligence (AmI) was defined as a built environment imbued with computing hardware and software technology to benefit its occupant(s) (Zelkha et al., 1998). More recently, architecture concerned with interaction between human and non-human agents has proliferated at various scales ranging from furniture to urban scale (Negroponte, 1975; Pask, 1975; Zelkha et al., 1998; Fox and Kemp, 2009; Fox, 2010; Oosterhuis and Bier, 2013; Bier and Knight, 2014; Robotic Building and Bier, 2018; Alavi et al., 2019; Pillan et al., 2020).

At furniture scale, examples such as The Big Data (2016) furniture and Media Block Chair (2012) are relevant as they demonstrate furniture reacting to people’s movement by changing color.¹ Such furniture takes advantage of location-based context-aware services and Internet connectivity that are ubiquitously available. Big Data furniture, for instance, communicates with users by changing colors in response to users’ movement. The movement and behavior of users as well as environmental data patterns are shared on an online database that is used to modulate user experience.

While these examples rely on sensor-actuator networks and involve some level of intelligence they do not involve AI relying on Computer Vision (CV)² for spatial reconfiguration as in the presented case study. With AI progressing and evolving in time through various stages of innovation and practical application, spatial reconfiguration is challenged to integrate multimodal technologies and improve HBI.

1.2 Contribution

The presented case study implemented with MSc students at Technical University (TU) Delft integrates AI-supported multimodal technologies into an alternative design approach relying on D2RPA&O methods. It addresses both the technical as well as architectural aspects from the early stages of the design process. It introduces as main features spatial reconfiguration relying on object recognition and lightweight material design. These are of relevance, particularly when the furniture consists, as in this case, of non-identical but similar furniture components that are re−/configured in a variety of possible combinations.

2 Implementation

Implemented with a group of architecture students tutored by experts from architecture, robotics, and computer science, as well as interaction and structural design, the project focused on the development of furniture components that offer reconfigurable indoor seating areas for accommodating various activities such as workshops (involving collaborative work sessions), events (requiring a stage and seating for the audience), group and individual studying activities next to temporary exhibitions taking place simultaneously or in short sequence. Hence spatial reconfiguration was paramount. Furthermore, the furniture was supposed to be taking structural, functional, environmental, and assembly considerations into account and have degrees of porosity, where the degree and distribution of porosity, i.e., density would facilitate both passive (structural strength, physical comfort, etc.) and active behaviors (interaction, etc.).

The approach involved D2RPA&O methods with the aim to improve spatial experience, process- and material-efficiency as well as embed intelligence in building processes and buildings by (a) computationally and robotically optimizing design and material production, (b) AI-supporting dis−/re−/assembly process, and (c) embedding sensor-actuators that enable interaction with users and the physically built environment. While these procedural steps were all implemented, they achieved various degrees of completion. For instance, the material production was implemented on a fragment, the AI model was tested on a synthetic data set, and the interaction was developed only conceptually at this stage. Future work will involve completing all steps in order to test the proposed varied and multiple use of the space in the library by allowing users to modify the configuration of furniture to meet their changing needs.

2.1 D2RPA&O

The D2RPA&O process relied on a Voronoi-based design that facilitates spatial reconfiguration by detaching and reattaching furniture components, i.e., booths to create various functional configurations. The challenge for users to identify availability of components and possible reconfiguration modes ranging from dispersed to clustered configurations that are serving varying activities has been addressed by means of interactive lighting, color coding, and AI-supported object recognition (Figure 1). Furthermore, the challenge for users to move components from one location to another has been addressed by implementing a highly porous, lightweight material design. All aspects were developed at conceptual level and achieved various degrees of implementation ranging from design to virtual and physical prototyping.

Figure 1

Figure 1. Red movable Voronoi-based component (left) and available component lighted green (left and right).

2.1.1 Conceptual design

The need for spatial reconfiguration has been identified by investigating user demands that were captured through a series of behavioral observations in the library space. These included the analysis of current occupancy patterns in terms of number of occupants and type of activities performed over one week. Additional unstructured observations were implemented ahead of the study by the students as they were themselves users of the library. Patterns of stagnation and movement were observed to understand how long users spend in various sections of the library and how users interact with others and various concerns were identified as for instance the reduced reconfigurability options of the furniture and lack of insight into available furniture meeting requirements for various activities and their location.

The occupancy pattern of the library showed large variability in terms of number of occupants during the year (with strong fluctuations in exam vs. vacation periods) and activities. However, even if the largest number of occupants were single users, the space was also used by groups of people for joint activities and one of the problems was the lack of furniture that could accommodate and adapt to user demands during respective activities. Reconfigurability and adaptability emerged therefore as the key objective of the presented design solution.

Reconfiguration has various challenges in terms of function, form, and requirements for lightweight. While precedents such as Media Block Chair presented interactive features, Cloud City, Office Excavate, FRGM: 1, and Voronoi Helmet³ showcased the potential of Voronoi-based design at building, component, and material scale. Hence, these approaches were adopted and enhanced using AI.

2.1.2 Computational design

Aspects of computational design were addressed by generating a Voronoi-based design at macro (i.e., building), meso (i.e., component), and micro (i.e., material) scales. This implies that the Voronoi-based material design of components and the assembly of components into a larger structure to facilitate spatial reconfiguration is scalable.

In order to optimize production and assembly of the furniture components, functional, structural, and material design studies were implemented. To optimize functional requirements in relation to activity patterns, reconfiguration by dis−/ reassembly of furniture components was chosen. To structurally optimize the Voronoi-based furniture components, a Karamba3D simulation that provided insight into the forces and deformations of the components, both for the isolated wireframe as for the surface elements (Figure 2) was implemented. The Voronoi-based material design with various cell densities is mapped onto the frame and surface elements according to the optimization results. The variable densities pattern is efficient in terms of the material used versus the resulting degree of stiffness of the object and weight reduction.

Figure 2

Figure 2. Structural design at component scale involved analysis of tension and compression (left and left-middle) and deformation and stiffness (right-middle and right).

The pattern is calculated based on the stress distribution (Figure 3) and the load cases that are used are the combination of the loads from the overall model and the forces on the component due to its own weight. The main aim of this study was to improve the mechanical performance of the components while keeping them as lightweight as possible. After all, excessive weight would considerably impact their movability.

Figure 3

Figure 3. Structural design at material scale without and with optimization (left and right, respectively) showing variation in density of Voronoi cells that are represented in varying colors on x and y coordinates in meters.

The considered material is thermoplastic elastomer (TPE) with a density of 1.1 g/cm3. The paneling geometry is thickened to 10 mm and based on this volume distributed loads are applied to the surfaces of the panels. Loads applied are line loads based on the weight of the panels. The resulting principal stress concentrations are translated into a point cloud with a higher density around the edges. The framework is thickened and calculated separately. Since the booths are freestanding simple supporting points are used for the structural analysis that creates a point cloud based on the relative stress densities. The point cloud is then transformed into a Voronoi pattern.

An additional structural analysis is run to determine the final values for the cell sizes and wireframe thicknesses. The Voronoi edges are thickened, and the volume is calculated. The applied loads are based on the weight of TPE and the volume of the wireframe structure. The structural analysis shows the deformation of the wireframe. The final thickness of the edges and cell size parameters are determined by a resulting deflection of less than 10 mm for each cell.

2.1.3 Interaction design

The primary goal to create a user-centered spatial experience and provide effective and adaptive spatial configurations to meet current user needs in the library includes consideration for accommodating changing use (single or group use, with various levels of privacy) and user-furniture interaction through responsive lighting in order to communicate users the potential availability of furniture components (Figure 1).

The user-light interaction system was conceptually designed to provide visual comfort and information to users. The use of indoor lighting for communicating with users while providing appropriate indoor visual conditions was developed and tested in previous work (Ackerly and Brager, 2013; Meerbeek et al., 2016; Luna-Navarro et al., 2020) and has been applied to this case study. While Light-emitting Diodes (LEDs) are integrated into the furniture to provide comfortable direct lighting, various, changing colors are used to communicate if it is available. Using an active system of pressure sensors integrated into the seats, the furniture responds to various types of use. Outward lighting signals availability, whereas inward lighting illuminates the occupied components while in use.

The challenges related to the integration of micro-controllers, sensors and actuators, i.e., LEDs into the furniture components involved ensuring that the physical and virtual subcomponents communicate data to the cloud and data is processed fast enough to allow for smooth uninterrupted interaction. This has been, however, only developed conceptually in this study, with the physical prototyping focusing on the materialization of variable stiffness with cushioning based on structural and ergonomic considerations.

2.1.4 Material design and robotic prototyping

Component and material design was Voronoi-based and involved consideration with respect to stability, lightweight, and comfort requiring variable stiffness and 3D printing without support.⁴ The variable stiffness concept developed in previous research (Hidding et al., 2019) has been implemented in this case study in order to create cushioning for the seating areas while keeping the structural integrity of the furniture components, i.e., booths intact. The cushioning principle was prototyped by robotically milling into Expanded Polystyrene (EPS) that in this case was emulating the support surface, which has high stiffness. On top of the support surface is the cushioning surface with low stiffness and is implemented by 3D printing with Thermoplastic Polyurethane (TPU). The imprint of the Voronoi-based pattern designed to serve as cushioning surface was milled out of the EPS so the 3D printed part could be fitted onto the surface (Figure 4).

Figure 4

Figure 4. Material design (left) and prototyped fragment (middle and right) using robotic subtractive and additive methods.

The variable stiffness design of the cushion is based on structural analysis, where the stiffness is controlled by the cell sizes implying that the smaller the cells, the denser the material and, therefore, the stiffer the structure. The design of the cells with their varying size and distribution depends on the structural analysis as well as the requirements for printing without support. In this context, the material design and prototyping approach relied on optimized material distribution based on structural optimization ensuring minimized material use and reduced production time. When compared with other production methodologies, the robotic prototyping of the Voronoi-based components has several major advantages. Firstly, it allows the production of complex geometries by linking the parametric design with the production data of the robotic equipment (D2RP). In the case of robotic 3D printing, the angles of the geometry have to be optimized to be within the angle constraints of −45 to +45 degrees for printing without support material (Hidding et al., 2019). Also, the size of the components and subcomponents are optimized to fit in the bounding box determined by the reachability of the robotic arm. The printability of the material, TPE, and its recyclability have to be considered as well.

2.2 Computer vision and artificial intelligence

Computer Vision (CV) as an Artificial Intelligence (AI) module is used for the identification of (a) detached furniture components and (b) the places where the components are supposed to be reattached (Figure 5). Specifically, a Deep Learning (DL) model⁵ is trained from scratch on a synthetic dataset. The input and output of the model are images, pointing for the input at a specific detached, i.e., stand-alone component, and for the output indicating its position and orientation in the constellation.

Figure 5

Figure 5. Object and placement recognition for the missing component (left), input image with the missing part (right).

2.2.1 Task description

The CV task was initially cast as a 3D puzzle reconstruction, where all the components are considered unique pieces of the constellation having their own individual base configuration. The problem could be formulated as a visual placement recognition for machine vision, given adequate training data with various configurations of missing pieces in the constellation (Figure 5). Even though an exciting CV challenge, the 3D puzzle reconstruction appeared out of the scope for this student project. The problem has then been reformulated to object recognition which is a well-known computer vision task with available model architectures. The components are labelled with ID numbers that are the keys to the correct placement in the constellation. Thus, the challenge is to predict the object’s ID number given an input image of the component taken by the user.

2.2.2 Dataset

A synthetic training and validation dataset is created to develop a deep learning model for the object detection task. Due to the specific configuration for this problem, no pre-trained model was available for effective deployment, hence, the students trained an object detection model from scratch to meet the requirements for this project. For the training set, Voronoi-based components are randomly distributed in a 3D model of the library (Figure 6), and around 3,000 images were generated from the scattered objects from different viewpoints. 80% of the data was used for training and 20% for evaluation. The synthetic dataset is curated by simulating a realistic scenario where the images are taken from real objects at the library. A generalization gap is, however, expected due to the distribution mismatch caused by lighting, shadowing, and occlusion, which are present in a real setting. This gap is increased by the differences between the 3D model and the 3D-printed elements.

Figure 6

Figure 6. Random camera and components distribution within the 3D model of library (top view left), an example input image from the setting (middle), and object recognition (right) implemented via an app on the phone.

The differences between the digital files and physical objects after 3D printing could be significant enough for the AI to perform poorly on photographs of these components. Creating 3D models that have the 3D-printed layers in the models could help to create training images that more closely resemble the produced components. Labelled photographs of the produced components could also be used to re-train the model in the case of poor performance. This has been, however, not the focus of this study and will be implemented in the next step.

2.2.3 Model and results

The Yolo object detection model (Redmon et al., 2016) is trained on the created dataset to predict the object’s ID number for the Voronoi-based components with acceptable accuracy and confidence. An online demo⁶ and screenshots from the demo (Figure 6) show randomly placed components that are detected in real time using the trained object detection model. The probability of specific components being present within the bounding box is shown in the right text box next to the box (Figure 6 right). These probabilities are used to classify an object as a true positive (correctly detected) or false positive (wrongly detected). Setting a confidence level as a hyperparameter for measuring the precision score for the object detection model. In this case, a confidence level of 0.95 from 1.00 has been achieved. There is a trade-off between precision and recall, therefore for such a high confidence level false negatives are increasing.

3 Discussion and future work

Presented case study outlines D2RPA&O strategies for AI-supported furniture reconfiguration as part of a larger HBI approach. Developed computational, interaction, material design, robotic production, and AI-supported assembly approaches (Figure 7) facilitate furniture adaptation to various functions and spatial requirements by allowing users to modify furniture configurations to meet changing needs in the multi-functional space of the TU Delft library.

Figure 7

Figure 7. Diagram showing the workflow of AI-supported assembly using an app on a phone to identify the components and their base placement location in real-time.

In the presented case study, individual furniture components were re−/ arranged to create various spatial configurations by moving, de−/ re−/ attaching, and/ or nesting together components. In this context, two main configurations, clustered (Figure 5) and dispersed (Figure 6), were considered. Additional solutions for individual cluster configurations will be investigated in the next iteration to identify the limitations of the system in terms of number of furniture components and their possible recombination in relation to the available space. Furthermore, by physically prototyping whole components and using photographs of the produced components to re-train the AI model the feasibility of this approach for real-life applications will be demonstrated.

In furniture components integrated interactive lighting was designed to facilitate user engagement via color- and intensity-changing LED lights that adjust to users’ preferences and communicate various states (un−/ available) and moods (open/ closed). In this context, HBI relied on the integration of technology, design, and user-centered approaches to create spaces that are responsive to the needs and behaviors of their occupants. It involved developing an understanding of the needs, preferences, and behaviors of the people who occupy the space. However, in order to make reconfiguration accessible to people of all abilities the robotization of the system is considered in the future.

4 Conclusion

While employed AI-supported approach presented in this paper is not new, its integration with D2RPA&O methods advances however current state-of-the-art in HBI mainly through the integrative multimodal approach. It contributes to the advancement of HBI by employing spatial reconfiguration relying on AI and lightweight material design, which is of relevance, particularly when the furniture consists of non-identical but similar components that are re−/ configured in a variety of possible combinations.

The D2RPA&O methods ensuring reduction of material use and production time via various optimization techniques are complemented by AI-supported spatial reconfiguration approaches with the ultimate goal of improving efficiency of space use and users’ comfort while decreasing environmental impact.

The evaluation and testing were implemented by computationally designing the structure, simulating the whole process and prototyping a fragment at 1:1 scale. Since the focus was on the integration of various research areas involving architecture, computer science, interaction, and structural design, the study met its objectives as all aspects were considered although D2RO remained at conceptual and computational design levels, while D2RP&A reached virtual and physical prototyping levels (Figures 7, 8).

Figure 8

Figure 8. Diagram showing D2RP process reaching prototyping level.

The use of lightweight material design for physical dis−/ reassembly and the use of lights to provide user-building interaction strategies by communicating to users which furniture components and when they are available were conceptually and to some degree physically explored, though, tests with users still need to be conducted in order to evaluate user response and satisfaction with proposed interaction strategy. Also, the use of AI on physically prototyped components needs verification and operability by users with variable physical abilities needs consideration, which will be addressed in the future.

The collaborative approach required the integration of all aspects from the very beginning of the process. While architecture focused on function and form meeting structural and material design requirements, the interaction aspects relied on the integration of CV. The project, however, did not get to the point of testing the CV approach on a physically prototyped furniture component missing to establish the feasibility of the CV to adequately operate in situ, i.e., under realistic conditions. In future, the feedback from the performance of the CV on the physically prototyped furniture component will be used to update the training of the algorithm. For legitimate research purposes the data set used in this study can be obtained by contacting the corresponding author.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://repository.tudelft.nl/.

Author contributions

HB: Writing – review & editing, Writing – original draft, Validation, Supervision, Project administration, Methodology, Investigation, Data curation, Conceptualization. AH: Writing – original draft, Validation, Software, Methodology, Investigation, Data curation. SB: Writing – original draft, Validation, Methodology, Investigation. AL-N: Writing – original draft, Methodology, Investigation. SK: Writing – original draft, Validation, Methodology, Investigation. CE: Writing – original draft, Validation, Methodology, Investigation.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

This paper has profited from the contribution of MSc 2 students Oliver Post, who developed the code, Jiacheng Xu, and Jiahui Shi, who applied the AI-supported D2RPA&O methods to the case study, under the supervision of tutors from the Robotic Building and AiDAPT labs as well as Structural Design and Mechanics and Design of Construction chairs. Presented project could not have been implemented without the support of the TU Delft Library (especially Open Spaces programmer, Vincent Cellucci) acting as client and co-founding partner.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Links to projects: https://www.bassala.com/big-data-furniture and https://www.teamlab.art/w/mediablockchair/.

2. ^AI is usually used to solve problems that typically require human intelligence such as learning, reasoning, problem-solving, perceiving, etc., while CV as a subfield of AI particularly focuses on interpreting visual data.

3. ^Links to Cloud City, Office Excavate, FRGM: 1, and Voronoi Helmet: https://www.metalocus.es/en/news/tomas-saraceno-roof-cloud-city-met, https://www.coroflot.com/GrahamKelman/Office-Excavate, http://www.goetz-schrader.com/crack01.html, and https://www.designboom.com/design/voronoi-bicycle-helmet-08-10-2020/.

4. ^This implies that the angles of the faces of the Voronoi cells have to be within 3D printing tolerances with printing angles between −45 to +45 degrees in relation to the printing bed.

5. ^Deep learning is a machine learning method, which is based on artificial neural networks that are built using principles of organization in the biological neural networks of animal brains (https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414).

6. ^Link to RB-page: http://www.roboticbuilding.eu/project/human-robot-interaction-for-d2ra/.

References

Ackerly, K., and Brager, G. (2013). Window signalling systems: control strategies and occupant behaviour. Build. Res. Inf. 41, 342–360. doi: 10.1080/09613218.2013.772044

Crossref Full Text | Google Scholar

Alavi, H., Churchill, E., Wiberg, M., Lalanne, D., Dalsgaard, P., Schieck, A., et al. (2019). Introduction to human-building interaction (HBI): interfacing HCI with architecture and Urban Design. ACM Trans. Comp. Human Interact. 26, 1–10. doi: 10.1145/3309714

Crossref Full Text | Google Scholar

Bier, H., and Knight, T. (2014). Data-driven design to production and operation. Footprint. 8, 1–8. doi: 10.7480/FOOTPRINT.8.2.807

Crossref Full Text | Google Scholar

Fox, M. Catching up with the past: a small contribution to a long history of interactive environments (2010).

Google Scholar

Fox, M., and Kemp, M. (2009). Interactive architecture. New York: Princeton Architectural Press.

Google Scholar

Hidding, A., Bier, H., Wang, Q., Teuffel, P., and Senatore, G. (2019). Structural adaptation through stiffness tuning. Spool. 6, 43–48. doi: 10.7480/SPOOL.2019.1.3892

Crossref Full Text | Google Scholar

Luna-Navarro, A., Loonen, R., Juaristi, M., Monge-Barrio, A., Attia, S., and Overend, M. (2020). Occupant-facade interaction: a review and classification scheme. Build. Environ. 177:106880. doi: 10.1016/j.buildenv.2020.106880

Crossref Full Text | Google Scholar

Meerbeek, B. W., de Bakker, C., de Kort, Y. A. W., van Loenen, E. J., and Bergman, T. (2016). Automated blinds with light feedback to increase occupant satisfaction and energy saving. Build. Environ. 103, 70–85. doi: 10.1016/j.buildenv.2016.04.002

Crossref Full Text | Google Scholar

Negroponte, N. (1975). Soft architecture machines. Cambridge, MA: MIT Press.

Google Scholar

Oosterhuis, K., and Bier, H. (2013). Robotics in architecture. Heijningen: Jap Sam Books.

Google Scholar

Pask, G. (1975). Artificial intelligence: A preface and a theory, soft architecture machines. Cambridge, MA: MIT Press.

Google Scholar

Pillan, M., Bier, H., Green, K., and Pavlovic, M. (2020). Actuated and performative architecture: emerging forms of human-machine interaction. Cyber Physical Architecture 7, 3–4. doi: 10.7480/SPOOL.2020.3

Crossref Full Text | Google Scholar

Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. You only look once: unified, real-time object detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: Las Vegas, NV, USA(2016).

Google Scholar

Robotic Building (2018). Springer series in adaptive environments. Cham: Springer International Publishing.

Google Scholar

Zelkha, E., Epstein, B., Birrell, S., and Dodsworth, C.. From devices to “Ambient Intelligence”: the transformation of consumer electronics. Digital living room conference (1998).

Google Scholar