User:Ukiriz/sandbox/Computational creativity 2

Overview: I added the Text-to-image Model subsection under the Machine learning for Computational creativity section. The added text is bolded and underlined.

Machine learning for Computational creativity

An image generated by a text-to-image model with the prompt photograph of an astronaut riding a horse on moon

While traditional computational approaches to creativity rely on the explicit formulation of prescriptions by developers and a certain degree of randomness in computer programs, machine learning methods allow computer programs to learn on heuristics from input data enabling creative capacities within the computer programs. Especially, deep artificial neural networks allow to learn patterns from input data that allow for the non-linear generation of creative artefacts. Before 1989, artificial neural networks have been used to model certain aspects of creativity. Peter Todd (1989) first trained a neural network to reproduce musical melodies from a training set of musical pieces. Then he used a change algorithm to modify the network's input parameters. The network was able to randomly generate new music in a highly uncontrolled manner. In 1992, Todd extended this work, using the so-called distal teacher approach that had been developed by Paul Munro, Paul Werbos, D. Nguyen and Bernard Widrow, Michael I. Jordan and David Rumelhart. In the new approach, there are two neural networks, one of which is supplying training patterns to another. In later efforts by Todd, a composer would select a set of melodies that define the melody space, position them on a 2-d plane with a mouse-based graphic interface, and train a connectionist network to produce those melodies, and listen to the new "interpolated" melodies that the network generates corresponding to intermediate points in the 2-d plane.

Language Models and Hallucination

Language models like GPT and LSTM are used to generate texts for creative purposes, such as novels and scripts. These models demonstrate hallucination from time to time, where erroneous materials are presented as factual. Creators make use of their hallucinatory tendency to capture unintended results. Ross Goodwin's 1 the Road, for example, uses an LSTM model trained on literature corpora to generate a novel that refers to Jack Kerouac's On the Road based on multimodal input captured by a camera, a microphone, a laptop's inner clock, and a GPS throughout the road trip. Brian Merchant commented on the novel as "pixelated poetry in its ragtag assemblage of modern American imagery". Oscar Sharp and Ross Goodwin created the experimental sci-fi short film Sunspring in 2016, written with an LSTM model, trained on their scripts and 1980-1990 sci-fi movies. Rodica Gotca critiqued their overall lack of focus on the narrative and intention to create based on the background of human culture.

Nevertheless, researchers highlight the positive side of language models' hallucination for generating novel solutions, given that the correctness and consistency of the response could be controlled. Jiang et al. propose the divergence-convergence flow model for harnessing the hallucinatory effects. They summarize the types of such effects in current research into factuality hallucinations and faithfulness hallucinations, which can be divided into smaller classes like factual fabrication and instruction inconsistency. While the divergence stage involves generating potentially hallucinatory content, the convergence stage focuses on filtering the hallucinations that are useful for the user with intent recognition and evaluation metrics.

Text-to-Image Model

Recent advancements in computational creativity in the image-text medium are represented by text-to-image models that enable the generation of digital artworks from natural language descriptions. These systems have evolved rapidly since OpenAI introduced CLIP (Contrastive Language-Image Pre-training) in January 2021, which learned from web-scale training data to associate visual concepts with textual descriptions. While early models like VQGAN-CLIP and BigSleep combine CLIP with GAN (Generative Adversarial Network), later ones like DALL-E, Midjourney, and Stable Diffusion incorporate diffusion models instead of GAN, which allows the production of more photo-realistic digital images from textual prompts. Nevertheless, Hannah Johnston points out the "surreal quality that has since been largely lost" due to this transition.

The creative potential of these systems lies not merely in their computational output, but in the human-AI collaborative process they facilitate. Zhou and Lee's research, analyzing over 4 million artworks from more than 50,000 users, shows that text-to-image AI significantly enhances human creative productivity by 25% and increases the value as measured by the likelihood of receiving a favorite per view by 50%. However, this productivity increase comes with nuanced effects on creativity itself. When top-quality work Content Novelty, or center piece theme and connections, increases over time, mean Content Novelty decreases, a sign that the resulting idea space is larger but wasteful. This suggests that some creators push boundaries of creativity further than others, but in general, there is coalescing into visually uniform and similarly themed paintings.

The process of artistic text-to-image creation relies on "prompt engineering" – the iterative task of crafting effective textual inputs to condition the AI model. It depends on understanding the training data of the model, experimenting with style modifiers, and engaging in what researchers have called "generative synesthesia" – the seamless blending of human discovery and AI leveraging to discover new creative pipelines. Jonas Oppenlaender emphasizes the entire process of human-AI interaction, including image-level and portfolio-level selection, over the final output of the text-to-image model. Referring to Mel Rhodes' four-P model (person, process, press, product) to reframe computational creativity, Oppenlaender highlights the entire creative pipeline – the practitioner, the iterative process, the community environment, and the chosen output – to evaluate AI-assisted art creation.

Online communities are now at the core of this system, serving as learning platforms as practitioners share techniques and prompts. These communities make high-level art production tools available to everyone, allowing creators who lack formal artistic education to produce high-quality visual material. AI-boosted artists capable of executing more innovative ideas without regard to their prior originality might produce works that are rated more favorably by peers.

The technology raises profound questions over authorship and creativity in the contemporary age. There are legal implications, such as when AI-generated content has won art competitions or artists asking for copyright on AI-assisted content. Such developments remind us of previous argument over the validity of photography as art, suggesting a paradigm shift over how society conceptualizes and perceives creative work.