3rd AI-ARTS Competition 2024

The competition format will be similar to previous years, with the addition of an AI judge accounting for 33.3% of the total evaluation. The evaluation will consist of three components:

1. Public Evaluation:

Method: Participants and viewers can give a thumbs-up to their favourite artworks.

Impact: This feedback reflects the popularity and appeal of the artworks to a broader audience.

2. Organisers’ Evaluation:

Method: A private, internal assessment conducted by the organisers.

Criteria: Includes various aspects such as the artist’s technique, originality, creativity, and the artist’s profile.

Impact: This evaluation provides a professional and experienced perspective on the artworks.

3. AI Judge Evaluation:

Method: An AI system will analyse the artworks based on predefined criteria including creativity, originality, technical execution, relevance of the theme, emotional impact, aesthetics, societal impact, and art experience.

Impact: The AI Judge aims to provide a balanced and unbiased evaluation by assessing all submissions using the same parameters, the same underlying AI model, and at the same stage in AI development. This consistent approach is intended to ensure fairness and objectivity in the assessment process (please refer to the “Constraints and Limitations” section below).

1. Submission

Artwork Submission:

Artists will submit their AI-generated artworks using the designated submission forms, similar to previous competitions.

Each submission can include up to a maximum of 20 images, plus one featured image and text fields for additional descriptions or context.

Publication and Voting:

Submitted artworks will be published on the competition platform.

The general public will be allowed to vote for their favourite artworks using the thumbs-up button during the voting period.

2. Evaluation:

Public Voting:

Votes from the public, collected via the thumbs-up button, will be counted and contribute to the overall score.

AI Judge Evaluation:

All submissions will be evaluated by the AI Judge, which will analyse the images and any accompanying text.

Organisers’ Evaluation:

All images will be evaluated by the competition organisers based on internal criteria.

Main Representative Image:

Artists must select one main representative image for extensive AI analysis. By default, this will be the featured image unless another image is specified by the artist. The submission form will include a field to select the representative image.

Any text included in the submission will also be considered during the AI evaluation.

Determining Winners:

Winners will be determined based on a combination of public votes, organisers’ evaluations, and AI Judge scores, each contributing equally to the final decision.

3. Feedback

AI Evaluation Report: At the end of the competition, each artist will receive a detailed AI evaluation of their submissions along with a short feedback form.

Public Assessment Option: Artists will have the option to make their AI scoring assessment public. If chosen, the AI scores will be published alongside the submission on the competition platform.

Final Report: A comprehensive report summarizing the feedback and outcomes of the experiment will be published on the competition website, providing insights into the viability and effectiveness of AI-assisted art evaluation.

The AI Judge will use state-of-the-art technologies to evaluate both images and text submitted in the competition. It leverages large language models (LLMs) and multimodal AI models to process and analyze visual and textual content, providing a comprehensive evaluation.

AI Judge Evaluation Criteria

The AI judge will assess each AI-generated artwork submission using the following criteria:

Creativity: The uniqueness and imagination demonstrated in the artwork.

Originality: The novelty and distinctiveness of the concept and execution.

Technical Execution: The skill and precision in the creation of the artwork.

Relevance of the Theme: How well the artwork aligns with the theme selected by the author.

Emotional Impact: The ability of the artwork to evoke emotions in the viewer.

Aesthetics: The visual appeal and artistic quality of the piece.

Societal Impact: The relevance and commentary on social issues presented by the artwork.

Experience: This factor evaluates the overall impact and cohesiveness of an artwork as a lived experience. It assesses how effectively the various elements of the artwork come together to create a unified and compelling encounter for the viewer. This includes the harmony between narrative and visuals, the depth and resonance of the theme, and the ability to provoke thought and evoke emotions. It considers whether the artwork succeeds in engaging the audience on multiple levels, creating a meaningful and memorable experience.

Underlying Technologies

Multimodal AI Models

We are currently evaluating the best options for Multimodal AI models to use in the competition. These models integrate natural language processing (NLP) with image recognition and generation capabilities, ensuring thorough and balanced evaluations.

Platforms Being Considered:

ChatGPT: Utilises large-scale language models to understand and generate human-like text. Additionally, it incorporates image recognition and analysis capabilities to aid in comprehensive multimodal evaluations.Gemini: An advanced AI model designed for multimodal tasks, combining language understanding with image generation to provide comprehensive analysis and evaluation.Microsoft Azure OpenAI Service: Provides access to various OpenAI models capable of multimodal tasks, offering robust tools and scalability for large applications.

One of these technologies will be selected to make the AI Judge available for the competition, ensuring effective and fair evaluations of all submitted artworks. However, given the rapid evolution of AI tools, we do not rule out the possibility of using another advanced AI model that becomes available in the coming months.

Evolution and Reliability

As AI technologies rapidly evolve, we commit to using the most reliable model available at the start of the evaluation process. All artworks will be evaluated using the same methods to ensure fairness and consistency in the assessment process.

In any experimental setup, especially one involving cutting-edge technology like AI, it’s essential to acknowledge the constraints and limitations that might affect the outcomes. Here, we outline the primary limitations of our AI artwork competition experiment:

Quantitative Evaluation: The AI Judge primarily uses quantitative metrics to assess artworks, which may not capture the full depth of artistic expression and nuance that a qualitative, human-based assessment might provide.

Lack of Discussion and Consensus: Unlike human judges, the AI Judge cannot engage in discussions or reach a consensus. This absence of collaborative evaluation may overlook diverse perspectives and the richness of interpretive dialogue.

Simplified Assessment Process: The AI evaluation might not capture the full complexity and nuance of artistic expression as a human judge would. The AI judge uses predefined criteria that may oversimplify the evaluation of diverse and intricate artworks.

Biases in AI: AI systems can inherit biases from their training data, which may affect their evaluation outcomes. These biases could influence the AI’s assessment in ways that are not immediately apparent, potentially leading to unfair evaluations.

Uniform Criteria Application: Using the same criteria for all artworks might not adequately account for the diverse styles, themes, and contexts presented by different artists. This uniformity can lead to limitations in recognising the unique aspects of each submission.

Current Technological Constraints: The capabilities of current AI models are limited by the technology available today. While they can perform impressive analyses, they are not yet capable of fully understanding and appreciating art in the same way humans do.

Contextual Understanding: AI judges may lack the contextual understanding that human judges have. This includes cultural, historical, and personal nuances that can be crucial in interpreting and evaluating art.

Lack of Emotional and Personal Insight: AI lacks the capacity for emotional resonance and personal insight, which are crucial in evaluating the emotive and subjective qualities of art.

Participant Awareness: Artists participating in this competition will be aware of these limitations and understand the current technological constraints. This awareness is crucial for setting realistic expectations and for contributing meaningful feedback to improve future iterations of the experiment.

Feedback collected from artists will be incredibly valuable in assessing the validity and reliability of AI evaluations. This feedback will help us refine our methodologies and enhance the overall fairness and effectiveness of AI-assisted art assessment.