It would be an understatement to say that artificial intelligence (AI) has shook the world over the past several months. Since the launch of ChatGPT at the end of lastNovember, we at CXperts challenged ourselves to see how AI can help our day-to-day work.
Within the world of conversion rate optimization (CRO), it’s typical for testing programs to outgrow simple tests like ones that can be built using a point-and-click visual editor. As testing programs mature they rely more and more – if not exclusively – on front-end developers to build more complex tests via variation or experiment-level code. As I thought about how AI will affect CRO, a question came to mind: can AI be your front-end developer? Let's find out.
The Experiment
For this experiment, I used the free version of ChatGPT since it was the AI service I was most familiar with. For the test website, we used none other than our own CXperts.io website. Lastly, I used AB Tasty as the testing platform to build the test.
For this experiment, I provided ChatGPT with prompts to change the appearance and functionality of on-page elements. In most of these prompts, I provided sections of the site’s code relevant to the element or section that I wished to change. The prompt provided to ChatGPT and the response it generated will be provided for each test.
Use Case 1: Simple Text Change
Goal: Change text for a specific element. Most testing platforms can do this without requiring code, but we wanted to see if it can handle a very simple request.
Element:
Prompt:
Please write me a javascript to replace the text of this H1 tag to read: The Experts in Digital Customer Experience
Certainly! Here is your JavaScript code properly escaped so it can be printed on the screen in your HTML:
<!-- Get the h1 element by its class name -->
<script>
var h1Element = document.querySelector('.h1-hero');
// Check if the h1 element exists
if (h1Element) {
// Replace the text content of the h1 element
h1Element.textContent = 'The Experts in Digital Customer Experience';
} else {
console.error('H1 element with class "h1-hero" not found.');
}
</script>
You can place this script in a <script> tag at the end of your HTML body or in an external JavaScript file included at the end of your HTML file. This script will find the <h1> element with the class "h1-hero" and replace its text content with "The Experts in Digital Customer Experience".
Results: 👍 Works!
This was a simple test so I wasn’t too surprised that ChatGPT was able to write the code for this text change. What was interesting in ChatGPT’s response was the inclusion of code to check if the element was on the page and add console logs when the H1 was not on the page.
Use Case 2: Find and Replace
Goal: Replace every instance of the “CXperts” with “CXperts Inc”. This is a step up in complexity than a simple text edit.
Prompt:
I have the word "CXperts" without the quotes on my page. I want to append the word "Inc." to every instance of the word "CXperts" on the page so that it reads "CXperts Inc". Can you write the javascript code to do this?
Answer:
Sure, you can achieve this by selecting all the text nodes in the HTML document and then replacing the instances of "CXperts" with "CXperts Inc.". Here's how you can do it using JavaScript:
```javascript
// Function to replace text in all text nodes
function replaceTextInNode(node) {
if (node.nodeType === Node.TEXT_NODE) {
node.textContent = node.textContent.replace(/\bCXperts\b/g, 'CXperts Inc');
} else if (node.nodeType === Node.ELEMENT_NODE) {
for (let i = 0; i < node.childNodes.length; i++) {
replaceTextInNode(node.childNodes[i]);
}
}
}
// Call the function with the document body
replaceTextInNode(document.body);
```
This code will traverse all the nodes in the HTML document, and for each text node, it will replace the text "CXperts" with "CXperts Inc". This ensures that all occurrences of "CXperts" are replaced regardless of where they are in the document.
Results: 👍 Works!
The code ChatGPT provided was a bit more complex than the previous example, but it worked perfectly. One unintended change the code introduced was the change in the “About CXperts” nav link. Out of curiosity, I asked ChatGPT to limit the find and replace to just the main body by giving it the main body’s div class:
You provided me a script previously to replace every instance of the text string "CXperts" to "CXperts Inc". Here is the code:
// Function to replace text in all text nodes
function replaceTextInNode(node) {
if (node.nodeType === Node.TEXT_NODE) {
node.textContent = node.textContent.replace(/\bCXperts\b/g, 'CXperts Inc');
} else if (node.nodeType === Node.ELEMENT_NODE) {
for (let i = 0; i < node.childNodes.length; i++) {
replaceTextInNode(node.childNodes[i]);
}
}
}
// Call the function with the document body
replaceTextInNode(document.body);
Can you edit this so that it does the find and replace within this div? <div class="site-wrapper">
And ChatGPT responded with the updated code:
Sure, you can achieve this by selecting all the text nodes in the HTML document and then replacing the instances of "CXperts" with "CXperts Inc.". Here's how you can do it using JavaScript:
```javascript
// Function to replace text in all text nodes
function replaceTextInNode(node) {
if (node.nodeType === Node.TEXT_NODE) {
node.textContent = node.textContent.replace(/\bCXperts\b/g, 'CXperts Inc');
} else if (node.nodeType === Node.ELEMENT_NODE) {
for (let i = 0; i < node.childNodes.length; i++) {
replaceTextInNode(node.childNodes[i]);
}
}
}
// Call the function with the document body
replaceTextInNode(document.body);
```
This code will traverse all the nodes in the HTML document, and for each text node, it will replace the text "CXperts" with "CXperts Inc". This ensures that all occurrences of "CXperts" are replaced regardless of where they are in the document.
This worked as well. The code ChatGPT wrote worked flawlessly to find and replace every instance of “CXperts” in the main content area.
Use Case 3: Hover Over Drop Down
Goal: To make it so that the L2 nav opens on hover not click
This one’s considerably harder than the previous two examples.
Prompt:
For this one, I’m going to have to provide ChatGPT with the HTML for the entire nav. I also specified which nav element has a dropdown menu by providing the elements id:
On my site's nav, my site's navigation L2 dropdown opens by clicking on it. However, I want to change this to open on hover. Here is the code for the entire nav:
The nav element with the dropdown menu have the id "menu-dropdown". Can you write me a javascript code to make this change?
Answer:
To make the dropdown menu open on hover instead of click, you can use JavaScript to add event listeners that trigger the opening and closing of the dropdown menu. Here's how you can do it:
```javascript
// Wait for the DOM content to be fully loaded
document.addEventListener('DOMContentLoaded', function() {
// Get all the dropdown toggle elements
const dropdownToggles = document.querySelectorAll('.dropdown-toggle');
// Add event listeners to each dropdown toggle
dropdownToggles.forEach(function(toggle) {
// Show dropdown menu on mouse enter
toggle.addEventListener('mouseenter', function() {
const dropdownList = toggle.nextElementSibling;
dropdownList.classList.add('w--open');
});
// Hide dropdown menu on mouse leave
toggle.addEventListener('mouseleave', function() {
const dropdownList = toggle.nextElementSibling;
dropdownList.classList.remove('w--open');
});
});
});
```
This code will listen for `mouseenter` and `mouseleave` events on each dropdown toggle. When the mouse enters a dropdown toggle, it will add the class `w--open` to the corresponding dropdown list, making it visible. When the mouse leaves the toggle, it will remove the `w--open` class, hiding the dropdown list.
Ensure that this JavaScript code is placed after the HTML elements have been loaded, either at the end of the `<body>` tag or wrapped in a `DOMContentLoaded` event listener.
Results: 👎 Didn’t work
I wasn’t that surprised this didn’t work. Even though I gave ChatGPT a lot of the site’s code, it’s still trying to provide a solution without knowing the site. Additionally, it doesn’t know what kind of styles are applied to these elements so odds were against the AI here. Still, I gave ChatGPT another chance. This time I gave ChatGPT the code of the element in it’s opened state hoping that this can increase the chance of it giving me the right code.
Second Attempt Prompt:
This didn't work. If it helps, here is what the code looks like when the "What We Do" menu dropdown is open:
Can you review the above and update the code you provided me?
Answer:
Thank you for providing the additional information. Based on the structure of your HTML and the inline styles, it seems like the dropdown toggle has the class `dropdown-toggle` and the dropdown list has the class `dropdown-list`.
Here's the updated JavaScript code to make the dropdown menu open on hover:
```javascript
// Wait for the DOM content to be fully loaded
document.addEventListener('DOMContentLoaded', function() {
// Get all the dropdown toggle elements
const dropdownToggles = document.querySelectorAll('.dropdown-toggle');
// Add event listeners to each dropdown toggle
dropdownToggles.forEach(function(toggle) {
// Show dropdown menu on mouse enter
toggle.addEventListener('mouseenter', function() {
const dropdownList = toggle.nextElementSibling;
dropdownList.classList.add('w--open');
});
// Hide dropdown menu on mouse leave
toggle.addEventListener('mouseleave', function() {
const dropdownList = toggle.nextElementSibling;
dropdownList.classList.remove('w--open');
});
// Hide dropdown menu on mouse leave from the dropdown list
toggle.nextElementSibling.addEventListener('mouseleave', function() {
this.classList.remove('w--open');
});
});
});
```
This code should now work with your HTML structure. It listens for mouse enter and mouse leave events on the dropdown toggle to show and hide the dropdown list accordingly. Additionally, it hides the dropdown list when the mouse leaves it.
Results: 👎 Didn’t work
I was hoping that showing what the dropdown menus open state code looks like would help but it didn’t. It is possible to continue prompting ChatGPT with clarifying information or code to help it find an answer but there’s no guarantee that will work. However, ChatGPT does have browser extensions that could crawl a live website. It may be possible to generate a working script if it has access to the full site.
Use Case 4: Sticky Navigation
Goal: Make the desktop site nav sticky.
Sticking with the nav, I wanted to see if ChatGPT can make the site’s global navigation to stay fixed at the top of the site.
Prompt:
On my site, my site nav on desktop isn't sticky. Can you help me make the header a sticky element? Here is the div that contains the entire site nav:
Can you review the above and update the code you provided me?
Answer:
To make the header sticky, you can use CSS to set the `position` property of the `.navbar` class to `fixed`. Here's how you can do it:
```css
.navbar {
position: fixed;
top: 0;
width: 100%;
z-index: 1000; /* adjust z-index as needed */
background-color: #fff; /* adjust background color as needed */
}
```
Add this CSS to your stylesheet or within a `<style>` tag in the `<head>` section of your HTML document. This will make the header stick to the top of the viewport as the user scrolls down the page. Adjust the `z-index` and `background-color` values according to your design needs.
Results: 👍 Works but needs adjustment
I was pleasantly surprised that this worked. It’s not perfect; there’s a gap between the nav and the top of the page where content can be seen when scrolled. However, ChatGPT did make the requested element sticky.
Use Case 5: Accordion Element
Goal: Create an accordion.
For this last test, we challenged ChatGPT to create an accordion element from scratch. Additionally, we are specifying where such element should be placed.
Inspiration:
Prompt:
My prompt is a bit messy but I provided a lot more details since we’re creating something completely new.
I'd like to create an HTML element. I want to create an accordion with 3 different headers that expand content below it. Please put dummy header text for each of the 3 headers and two sentences of dummy text under each accordion. The accordion should expand on clicks to the entire header and should close when clicking on the header again. The text should be left justified. On the header, I'd like an SVG "+" symbol when it's in the closed state and a "-" when the accordion is expanded. Make the accordion 800 pixels wide max. The header font size should be 20 with a bold weight. The content in the accordion should be 16.
Answer:
Sure, here's an example of how you can create the accordion using HTML, CSS, and JavaScript:
This code creates an accordion with three headers, each containing some dummy text. Clicking on a header expands or collapses its content. The "+" symbol changes to "-" when the accordion is expanded.
Results: 👎 Didn’t work
ChatGPT provided a full HTML code so I just took the relevant code and replaced an element on the site with this code. I was excited at first because it looked right, but clicking on each of the header didn’t expand the accordion. This was still promising so I decided to go back to ChatGPT to troubleshoot.
Second Attempt Prompt:
It looks great but when I click on the header it didn't expand the accordion. Can you fix this?
Second Attempt Answer:
Apologies for the oversight. Let's add the functionality to expand and collapse the accordion when clicking on the header::
Now, clicking on the header will expand or collapse the accordion content accordingly.
Results: 👎 Didn’t work
The code still didn’t work. Interestingly, ChatGPT added additional styling to the accordion which was a nice touch. I didn’t want to give up so I tried one last time to make it work.
Third Attempt Prompt:
Note: I asked Chat GPT to write me a javascript of the previous code so that it injects the HTML below the H1 tag at the top of the page.
When I click on the accordion header, nothing happens. Why wouldn't this work?
Third Attempt Response:
Apologies for the oversight. It seems the event listeners were not properly attached to the accordion headers. Let's adjust the script to ensure the event listeners are attached correctly:
```javascript
// Create a new div element for the accordion
var accordionContainer = document.createElement('div');
accordionContainer.className = 'accordion';
// Create accordion items
var accordionItems = [
{ header: 'Header 1', content: ['Lorem ipsum dolor sit amet, consectetur adipiscing elit.', 'Nulla nec augue non augue suscipit auctor ut quis leo.'] },
{ header: 'Header 2', content: ['Phasellus eu orci eget magna mattis consectetur.', 'Vestibulum auctor neque eu lectus commodo, quis faucibus orci eleifend.'] },
{ header: 'Header 3', content: ['Integer nec libero at sapien convallis ullamcorper.', 'Fusce placerat libero sit amet eros tincidunt, id consequat ipsum cursus.'] }
];
// Loop through accordion items and create HTML elements
accordionItems.forEach(item => {
var accordionItem = document.createElement('div');
accordionItem.className = 'accordion-item';
var accordionHeader = document.createElement('div');
accordionHeader.className = 'accordion-header';
accordionHeader.textContent = item.header;
var accordionHeaderSvg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
accordionHeaderSvg.setAttribute('xmlns', 'http://www.w3.org/2000/svg');
accordionHeaderSvg.setAttribute('width', '16');
accordionHeaderSvg.setAttribute('height', '16');
accordionHeaderSvg.setAttribute('fill', 'currentColor');
accordionHeaderSvg.setAttribute('class', 'bi bi-plus');
accordionHeaderSvg.setAttribute('viewBox', '0 0 16 16');
var path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
path.setAttribute('d', 'M7.5 1.5a.5.5 0 0 1 .5.5V7h5a.5.5 0 1 1 0 1H8v5.5a.5.5 0 1 1-1 0V8H1a.5.5 0 0 1 0-1h5V2a.5.5 0 0 1 .5-.5z');
accordionHeaderSvg.appendChild(path);
accordionHeader.appendChild(accordionHeaderSvg);
var accordionContent = document.createElement('div');
accordionContent.className = 'accordion-content';
item.content.forEach(text => {
var paragraph = document.createElement('p');
paragraph.textContent = text;
accordionContent.appendChild(paragraph);
});
accordionItem.appendChild(accordionHeader);
accordionItem.appendChild(accordionContent);
accordionContainer.appendChild(accordionItem);
// Add event listener to accordion header
accordionHeader.addEventListener('click', function() {
accordionItem.classList.toggle('accordion-expanded');
var accordionIcon = accordionHeader.querySelector('svg');
accordionIcon.classList.toggle('bi-minus');
accordionIcon.classList.toggle('bi-plus');
accordionContent.classList.toggle('accordion-expanded');
});
});
// Find the reference element
var referenceElement = document.querySelector('.h1-hero');
// Insert the accordion container after the reference element
referenceElement.parentNode.insertBefore(accordionContainer, referenceElement.nextSibling);
```
Now, clicking on the header will expand or collapse the accordion content accordingly.
Results: 👍 It worked!
It took a few attempts but this accordion now works flawlessly. Oddly, some of my previous styling directions were not followed such as the text being left-aligned and have the plus “+” sign turn into a minus “-” sign when expanded. These cosmetic changes can be requested on a follow-up prompt.
Conclusion
It is clear from the tests that we conducted that you can create some simple to moderately complex experiences through ChatGPT. It’s totally feasible for a CRO with very light coding experience (like myself) to make codes from AI work.
However, I wouldn’t feel comfortable relying solely on AI at this point to build test experiences. ChatGPT was able to knock out simple requests but it required multiple prompts to get the codebase just right.
Additionally, ChatGPT is not consistent. Depending on the time of day when demand is higher, we have personally noticed ChatGPT responses to be more terse and error prone (similar to a human being).
Lastly, the most important thing that a front-end developer can do that ChatGPT can’t is being a consultant. The nature of ChatGPT is that it will keep trying to find you a solution even if it is impossible. A real dev would be able to talk you through the challenges of the build, why it can and can’t work, or set proper expectations. Consultation is invaluable when designing and building a test and ChatGPT is just not quite there (yet).
.png){kind=logo}
