Week 4 (b): Sketch for My Project Visual Output

Initial Concept: Bioelectrical Data to MIDI-Driven Visuals

Initially, my idea was to convert a plant’s bioelectrical data into MIDI notes, which would then produce corresponding sound frequencies. These sound frequencies would be transmitted into TouchDesigner, influencing a 3d plant model’s particle dynamics. The goal was to create a visual representation of a plant’s biological activity, allowing the audience to perceive the plant’s signals in an immersive way.

1. Starting the visual Sketch for my virtual plant and particle Effect

To better understand the form and visual qualities I wanted for the particle effects and 3d modelling, I created a hand-drawn sketch of the plant (Fig. 1). This process helped me clarify the overall direction of the virtual plant’s structure and motion, especially how it might evolve during the TD visual effect.

The hand-drawn sketch allowed me to explore the plant’s rhythm and potential for interaction in a more intuitive way. I focused on combining characteristics from Evening Primrose (Oenothera). The Evening Primrose’s ability to adjust its nectar sweetness in response to environmental changes inspired me to incorporate environmental responsiveness into the visual behaviour of the virtual plant.

2. Generating Video from my sketch Images

After completing my hand-drawn sketches, I decided to convert the images into video sequences to further explore the plant’s potential for movement and transformation. For this process, I used RunwayML, a tool I became familiar with during the Arts and AI course. This allowed me to efficiently bring static visuals to life and experiment with how the plant might grow or react over time.

Using video as a format helps me better visualise the plant’s agency—how it might behave in a dynamic, responsive virtual environment. It also gives me a clearer sense of the direction for modelling the final virtual plant in 3d. Based on my sketch, I created two different video versions, each exploring a slightly different interpretation of the plant’s motion and form. This comparison helps me more clearly define the visual qualities I want to carry forward into the next stages of the project.

3. Transforming Video into Particle-Based Representation

After generating the videos using RunwayML, I imported them into TouchDesigner to simulate the virtual plant’s particle behavior. Having previously worked with TouchDesigner, I was already familiar with its particle system workflow, which allowed me to quickly create a visual mockup of how the plant might behave in a dynamic environment.

In line with my original concept, my goal was to break down the plant’s structure into flowing, organic particles that respond to its bioelectrical activity. This transformation suggests a plant that is not static, but continuously evolving within a digital ecosystem. In this setup, the particles are influenced by real-time MIDI sound data generated from the plant’s bioelectrical signals, allowing the virtual plant to visually reflect its physiological state.

Within TouchDesigner, I tested both versions of the animated videos. Personally, I preferred the particle behavior inspired by the second video. Unlike the first version, where particles clustered closely around the plant’s form, the second video showed particles moving and dispersing irregularly across space. This irregularity resonates more with my interpretation of plant bioelectrical signals—as non-linear and constantly shifting. To me, this unpredictable motion better expresses the living nature of the plant, allowing its virtual presence to feel more active and responsive in the digital environment.

This mockup process gave me an early sense of the visual language I want to develop. It helped me identify key elements such as particle irregularity, dispersion patterns, and their connection to bioelectrical data.