The competition format will be similar to previous years, with the addition of an AI judge accounting for 33.3% of the total evaluation. The evaluation will consist of three components:

1. Public Evaluation:

Method: Participants and viewers can give a thumbs-up to their favourite artworks.

Impact: This feedback reflects the popularity and appeal of the artworks to a broader audience.

2. Organisers’ Evaluation:

Method: A private, internal assessment conducted by the organisers.

Criteria: Includes various aspects such as the artist’s technique, originality, creativity, and the artist’s profile.

Impact: This evaluation provides a professional and experienced perspective on the artworks.

3. AI Judge Evaluation:

Method: An AI system will analyse the artworks based on predefined criteria including creativity, originality, technical execution, relevance of the theme, emotional impact, aesthetics, societal impact, and art experience.

Impact: The AI Judge aims to provide a balanced and unbiased evaluation by assessing all submissions using the same parameters, the same underlying AI model, and at the same stage in AI development. This consistent approach is intended to ensure fairness and objectivity in the assessment process (please refer to the “Constraints and Limitations” section below).

1. Submission

Artwork Submission:

Artists will submit their AI-generated artworks using the designated submission forms, similar to previous competitions.

Each submission can include up to a maximum of 20 images, plus one featured image and text fields for additional descriptions or context.

Publication and Voting:

Submitted artworks will be published on the competition platform.

The general public will be allowed to vote for their favourite artworks using the thumbs-up button during the voting period.

2. Evaluation:

Public Voting:

Votes from the public, collected via the thumbs-up button, will be counted and contribute to the overall score.

AI Judge Evaluation:

All submissions will be evaluated by the AI Judge, which will analyse the images and any accompanying text.

Organisers’ Evaluation:

All images will be evaluated by the competition organisers based on internal criteria.

Main Representative Image:

Artists must select one main representative image for extensive AI analysis. By default, this will be the featured image unless another image is specified by the artist. The submission form will include a field to select the representative image.

Any text included in the submission will also be considered during the AI evaluation.

Determining Winners:

Winners will be determined based on a combination of public votes, organisers’ evaluations, and AI Judge scores, each contributing equally to the final decision.

3. Feedback

AI Evaluation Report: At the end of the competition, each artist will receive a detailed AI evaluation of their submissions along with a short feedback form.

Public Assessment Option: Artists will have the option to make their AI scoring assessment public. If chosen, the AI scores will be published alongside the submission on the competition platform.

Final Report: A comprehensive report summarizing the feedback and outcomes of the experiment will be published on the competition website, providing insights into the viability and effectiveness of AI-assisted art evaluation.

The AI Judge will use state-of-the-art technologies to evaluate both images and text submitted in the competition. It leverages large language models (LLMs) and multimodal AI models to process and analyze visual and textual content, providing a comprehensive evaluation.

AI Judge Evaluation Criteria

The AI judge will assess each AI-generated artwork submission using the following criteria:

Creativity: The uniqueness and imagination demonstrated in the artwork.

Originality: The novelty and distinctiveness of the concept and execution.

Technical Execution: The skill and precision in the creation of the artwork.

Relevance of the Theme: How well the artwork aligns with the theme selected by the author.

Emotional Impact: The ability of the artwork to evoke emotions in the viewer.

Aesthetics: The visual appeal and artistic quality of the piece.

Societal Impact: The relevance and commentary on social issues presented by the artwork.

Experience: This factor evaluates the overall impact and cohesiveness of an artwork as a lived experience. It assesses how effectively the various elements of the artwork come together to create a unified and compelling encounter for the viewer. This includes the harmony between narrative and visuals, the depth and resonance of the theme, and the ability to provoke thought and evoke emotions. It considers whether the artwork succeeds in engaging the audience on multiple levels, creating a meaningful and memorable experience.

In any experimental setup, especially one involving cutting-edge technology like AI, it’s essential to acknowledge the constraints and limitations that might affect the outcomes. Here, we outline the primary limitations of our AI artwork competition experiment:

Quantitative Evaluation: The AI Judge primarily uses quantitative metrics to assess artworks, which may not capture the full depth of artistic expression and nuance that a qualitative, human-based assessment might provide.

Lack of Discussion and Consensus: Unlike human judges, the AI Judge cannot engage in discussions or reach a consensus. This absence of collaborative evaluation may overlook diverse perspectives and the richness of interpretive dialogue.

Simplified Assessment Process: The AI evaluation might not capture the full complexity and nuance of artistic expression as a human judge would. The AI judge uses predefined criteria that may oversimplify the evaluation of diverse and intricate artworks.

Biases in AI: AI systems can inherit biases from their training data, which may affect their evaluation outcomes. These biases could influence the AI’s assessment in ways that are not immediately apparent, potentially leading to unfair evaluations.

Uniform Criteria Application: Using the same criteria for all artworks might not adequately account for the diverse styles, themes, and contexts presented by different artists. This uniformity can lead to limitations in recognising the unique aspects of each submission.

Current Technological Constraints: The capabilities of current AI models are limited by the technology available today. While they can perform impressive analyses, they are not yet capable of fully understanding and appreciating art in the same way humans do.

Contextual Understanding: AI judges may lack the contextual understanding that human judges have. This includes cultural, historical, and personal nuances that can be crucial in interpreting and evaluating art.

Lack of Emotional and Personal Insight: AI lacks the capacity for emotional resonance and personal insight, which are crucial in evaluating the emotive and subjective qualities of art.

Participant Awareness: Artists participating in this competition will be aware of these limitations and understand the current technological constraints. This awareness is crucial for setting realistic expectations and for contributing meaningful feedback to improve future iterations of the experiment.

Feedback collected from artists will be incredibly valuable in assessing the validity and reliability of AI evaluations. This feedback will help us refine our methodologies and enhance the overall fairness and effectiveness of AI-assisted art assessment.