I Vibe Coded Synthetic Tests — this is how it went

Part 2 of building an AI-assisted pipeline for a real-world web app

Author’s note: this article was written with the assistance of ChatGPT. Also, as the field of AI assisted coding and debugging is moving very fast, this article could be outdated in 6–9 months.

Introduction

This article is a follow-up to “I Vibe Coded a Web App — and Was Genuinely Impressed”. In Part 1 of this series, I covered vibe coding a mock server and API tests. In this installment I cover creating simple synthetic monitors.

Code repo: simple-books-store-pipeline under the monitoring folder.

Building synthetic monitoring tests

Why this matters?

Synthetic monitoring is a technique used to simulate user interactions with a website, application, or API in order to proactively monitor performance and availability. It uses scripted, automated tests that mimic real user behavior from various locations and devices. These synthetic “users” perform actions such as:

• Visiting a web page
• Logging in
• Adding items to a cart
• Calling an API endpoint

Think of this as a “safety net” so that you have a set of tests that are always running so if there’s a bad production deploy or some 3rd party API starts failing (happens all the time) or one or more of your internal backends starts failing, you are able to find out about it before your customers do.

I highly recommend Checkly for synthetic monitoring because they are Playwright native. That means, unlike some other platforms, you could repurpose your existing Playwright tests to become synthetic monitors. They are also very git-friendly which just groks a lot better with dev teams (instead of having the overhead of QA or DevOps engineers working on synthetic monitoring).

Initial set up

First step is to get Cursor to index the checkly docs — see this for steps on how to do that.

Add https://www.checklyhq.com/docs/

Prompt:

create a folder on the top level called monitoring

Cursor created the “monitoring” folder

Prompt:

I want to create synthetic monitoring tests using checkly

Cursor asked to install: npm install -g @checkly/cli

Then there was some versioning issues that were disconnected with the documentation so Cursor was troubleshooting but it eventually was set up.

Also Cursor figured to use the simple API status checkand added it in as a first monitoring check (impressive)

Next I tried to run the status check locally:

npx checkly test

That failed cos I need to set up the API key. Naturally I asked Cursor how to do that and it guided me thru the steps.

This is where things started to go south.

First, I got the following error:

UnsupportedFileLoaderError: JitiFileLoader is not supported.

Cursor was consulted and suggested a few paths to try:

1. Modify the checkly.config.ts to checkly.condig.js and modify the code accordingly
2. Downgrade my version of Node (was 22) to 20.
3. Install latest version of Checkly

I tried all 3 and they all did not work 😫.

Author’s note: if you’re an experienced engineer you will know this is the time to take a break and take a step back. Your subconscious will then do its job to replay what happened and come up with possible paths forward. This is something AI can’t help you with (at least not now).

The breakthrough

After returning to this I tried using Claude to get another opinion. It gave me similar possible reasons for the failure except for

NODE_OPTIONS="--loader=ts-node/esm" checkly deploy

This I tried and got the error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ts-node' imported

That got me thinking that maybe Cursor didn’t have all the information (even tho we did index the Checkly docs in the beginning). I took a look at the Checkly docs site and found this:

I didn’t remember running those commands. So I asked Cursor to review that page and it immediately knew it had to set up Checkly correctly. The checkly.config.ts was recreated and via a series of other modifications (including installing jiti), we were able to get:

And once I got npx checkly deploy done:

The basic API status check on ChecklyHQ.com

Author’s note: I’ve reported this issue to the folks at Checkly and they are taking this feedback to make vibe coding using their tool as seamless as possible.

Converting an existing Playwright test to Checkly

Next I tried to convert an existing Playwright API test to Checkly test. The existing “should get list of books” test can be found in books-api.spec.ts under tests/api.

Prompt:

Can you convert the “should get list of books” Playwright API test in books-api.spec.ts in the “tests/api” folder to a checkly test?

Cursor converted the Playwright test into a Checkly test under the monitoring/__checks__ folder. Next was to test it out:

npx checkly test

There were a bunch of errors. Apparently the Playwright code generated by Cursor earlier did not grok with the accepted assertion syntax used by Checkly. Those were modified and checkly test was run again.

More errors.

Apparently methods like .matches, .isString() and .isBoolean() do not exist on the AssertionBuilder in the current Checkly SDK version. The correct approach is to use the available assertion methods, which are typically .equals(), .greaterThan(), .lessThan()and .contains()for JSON body assertions.

I told Cursor to make the changes accordingly and after a couple more minor modifications, npx checkly test worked. The next step was to deploy the Checkly test using npx checkly deploy.

The Books List Check on ChecklyHQ.com

Final thoughts

This was a great exercise to probe what Cursor is able to do in its current state. Due to the underlying model likely not having indexed some new information (possibly?) it was not able to provide the right guidance initially. To overcome this requires experience on what to do next. Most experienced engineers have hit this rut before and hence are able to recover from it but how about new engineers? How will new engineers get the exposure and/or guidance from experienced engineers if they are too dependent on AI tools? Perhaps it is a non issue cos these AI tools get better? Only time will tell.