Mental Palace:

I. Introduction

In this era of information overcrowd and social media, we’ve been living in this world with events happening everywhere. To store these events that are important to us, we will use cameras and phones to record those meaningful memories, so we can review them and feel those moments from time to time. However, there lies a truth that inevitably people tend to forget. Although video and other recording technologies present opportunities for storing and sharing memory, they also pose significant new vulnerabilities. When we watch videos from 14-inch screens, we only see a specific angle of what happened around us, which is a limited source of reconstructing memory. Additionally, with development of 3D building computational tool, in the future, people tend to be exposed to more 3D digital construction than 2D information. Sooner or later, two-dimensional external interferes will have less influence on mental information regeneration than three-dimensional external intrigues. In terms of better help people reconstruct and store memory, this project hypothesizes a new system, named Mental Palace, which affords an interactive environment for multiple users to input their memory as clips of videos, incorporate every detail of specific moment, reconstruct it in a 3D space and continuously adjust to the closest degree to the user's memory, based on users’ feedback.

II. Current Technology

In the fields of memory extraction and reconstruction, there are already some breakthroughs. Early in 2011, Professor Jack Gallant in UCB published research about developing a system to capture visual activity in human brains and reconstruct it as digital video clips. In that experiment, researchers used a special voxel system to decode the shape and motion information from the movies to specific brain actions and feed the computer with brain signals generated by moving pictures. Computer could generate a series of low-quality videos that were similar to the contents subjects saw. Given the exponential increase in computing power and our understanding of human biology, it may be no longer a movie to have people’s memory extracted and red by a machine. Also there exists a technology that can synchronize videos with similar patterns and then generate a longer video with full camera viewpoints, which is suitable for being used in memory videos incorporation. When it comes to 2D videos being transferred into 3D modeling, existing research shows that a novel framework named NeuralRecon makes this transformation achievable. This technology can detect and capture local smoothness prior and global shape prior of 3D surfaces when 3D model applications sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction.

III. Mental Canvas workflow

Based on current existing technologies, I hypothesize a new system called Mental Palace, that allows multiple users to extract their memory as clips of videos and input them into video synchronization technology, then a 3D virtual memory model world will be generated out of it, which will afford users to view this world as a third person perspective. In this project, people do not need to be nearly 100 percent correct about their memory sketch input, which is too hard and not an intendance of this project. People are allowed to express their feelings, mood, and emotions in the memory sketch because it is human’s different perspectives toward same events that makes this memory unique and valuable. Maybe in the wedding day of yours, you saw some guests smiling which makes you feel “pink” and “heart shape” about wedding scenarios. Others may feel the different way in different personal view. Combing in those different memory story from multiple angles, the results of this 3D space store each person's special story and past experiences that makes it more unique.

IV. Prototype

For the prototype of this project, I mainly built an interactive memory canvas in Processing and a sharing web called Mental Gallery in Visual studio code. In the Mental Canvas, users can type in the keyword of their memory, such as love, childhood, dream, or summer vacation in 2006. And then they can use the patterns that are listed on the left and the colors to customize the bricks. The patterns and colors should represent their feelings or impressions on this specific memory. After customization, users will enter the canvas and drag unlimited bricks to create their own unique work of their memory. The location of each brick is on users’ will. The result of final work will be the representative of users’ memory. Then users can save the framework and upload it from the default folder. Using Mental Gallery as a sharing platform, memory will have more meaning for bring happiness in life. What’s more, in the future, multiple users can even collage their memory and create a new clash of stories that will generate new sparkle of interaction between humankind.

V. Further implications

First for sure, if this technology is achievable in the future, it can not only help people reconstruct, store, and share their memory, but also strengthen their ability to value things in life and people that are meant so much for them. Additionally, in my perspective, technology should be consolation, that can smooth our tiredness out and bring peace in our life. Everyone, I believe, has their own regrets in life and things they don’t want to forget forever. Memory reconstruction is kind of a right path to solve that problem and somehow erase some of the sorrow, regrets and negative emotions. However, there still exist some problem remained to be solved, such as different recognition of the same thing due to age, experience of different ages, subjective influence on memory, and variety of sense that can’t be compressed into digital visualization, etc. But we must keep this in mind that reconstruct memory does not equal to rebuild history or reality, which is to say that it should be slightly different than what actually happened in the past. Bringing memory back to life can cause many social, ethical, and philosophical problems that would never be settled down. Therefore, rather than debating on whether to bring exact memory alive and arguing if it’s creating multiple-dimension universe, we can show the assistant ability of memory reconstruction and bring it into beneficial use in life. When I started this project, I’ve always been wondering if this project can be used into medication or recovery program of memory loss, traumatic recovery, and Alzheimer diseases. I believe that in the future, this technology could be benificial to humankind.

References

1. Nishimoto S, Gallant JL. A three-dimensional spatiotemporal receptive field model explains responses of area MT neurons to naturalistic movies. The Journal of Neuroscience : the Official Journal of the Society For Neuroscience. 31: 14551-64. PMID 21994372 DOI: 10.1523/JNEUROSCI.6801-10.2011
2. Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, Gallant JL. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology : Cb. 21: 1641-6. PMID 21945275 DOI: 10.1016/J.Cub.2011.08.031
3. Sun, Jiaming and Xie, Yiming and Chen, Linghao and Zhou, Xiaowei and Bao, Hujun. NeuralRecon : Real-Time Coherent {3D} Reconstruction from Monocular Video. CVPR 2021
4. Liang, Junwei; Burger, Susanne; Hauptmann, Alex; Aronson, Jay (2018): Video Synchronization and Sound Search for Human Rights Documentation and Conflict Monitoring. Carnegie Mellon University. Journal contribution.