Search for a command to run...
I described what to test in plain English & Passmark + Playwright handled the rest. Here’s what I found while testing eSewa — Nepal’s most popular digital wallet.
Software Developer 👩💻 | Technical Writer 📝 | Student 🎓
No comments yet. Be the first to comment.
The story starts after I finished eighth grade in late 2018. I came to know that the Ministry of Education runs the various Technical stream courses from grade 9 to grade 12. Among all of those, Computer Engineering was also one. This is not a course...
🎨 Got an idea, a challenge, or just a rough thought? 🌟 Introducing the revolutionary AI CEO!Just grab your virtual pen and draw your thoughts on a blank canvas. Whether it's a business challenge, a creative idea, or a complex dilemma, the AI CEO wi...
Effortless Article Sharing: Exploring the Power of 'git push' with gitHash on Hashnode!
In the series of debugging Feb, I am explaining the different types of errors that I encountered while working on a different project which ultimately became the reason to stay long hours in front of my laptop screen. Wrapping up the project was the ...
Can AI test Nepal’s biggest fintech platform using plain English? I tried.
Modern web applications move fast, but testing often struggles to keep up. A small UI update, renamed button, or layout change can break traditional automation and create unnecessary maintenance work.
That’s where things get interesting.
What if testing didn’t require writing complex scripts at all? What if we could just describe what we want to test… in plain English?
That’s exactly what Passmark enables.
For this experiment, I chose eSewa, one of Nepal’s most widely used digital wallet platforms trusted for payments, transfers, and everyday transactions. To make it happen, I used Passmark for plain-English AI testing flows and Playwright for browser automation.
In case you want to view the demo video directly. 👉 Demo Video
Traditional browser automation usually requires writing detailed selectors and maintaining fragile UI scripts. Even a small UI change can break the test flow.
For example, in a traditional Playwright test, interactions often look like this:
await page.locator('#login-btn').click();
await expect(page.locator('.login-modal')).toBeVisible();
This works, but it tightly depends on selectors, element structure, and implementation details.
Passmark changes this approach by allowing tests to be written in plain English using AI driven execution.
For example, while testing eSewa, I could simply write:
steps: [
{ description: "Navigate to https://esewa.com.np" },
{ description: "Click on the Login button" },
],
assertions: [
{ assertion: "Login modal is visible" },
{ assertion: "Phone number and password fields are present" },
]
Instead of manually targeting selectors, Passmark interprets the intent of the test case, while Playwright handles the browser automation .
When selecting a target for this hackathon, I wanted to work on something practical and meaningful.
The hackathon suggested testing any web app, including demo applications, which is great for experimentation. However, I decided to apply Passmark to a real-world use case instead.
That’s why I chose eSewa, one of Nepal’s most widely used digital wallet platforms. Since it handles real payments, transfers, and daily transactions, it represents a more realistic testing environment.
This made the experiment more interesting: How well can AI-powered regression testing handle a real production fintech system?
That question became the motivation for this project.
This project is a regression testing suite built using Passmark + Playwright to test real user flows of eSewa.
Instead of writing traditional automation scripts with selectors or page objects, I used Passmark to describe test steps in plain English, and Playwright to execute them in a real browser environment. The goal was to validate whether AI-driven testing can reliably handle real-world application flows such as navigation, UI interactions, and basic user journeys.
The test suite focuses on core user facing flows, including:
Landing page accessibility and navigation
Key UI interactions and button behavior
Basic user flow validation across pages
Regression checks to ensure stability across updates
This project was built using a modern testing stack centered around Passmark.
Passmark : This is AI driven testing framework that interprets natural language test steps and manages execution workflows.
Playwright : This is used under the hood for browser automation, navigation, and UI interactions.
OpenRouter API : This provides the AI model gateway for Passmark during test execution
TypeScript : This is used for writing structured and maintainable test files.
Node.js : Provide a runtime environment for running the test suite.
GitHub : This is used for version control.
In simple terms, Passmark handled the test intent, Playwright handled the browser actions, and the AI layer helped to bridge these two. This made the stack powerful while still keeping the developer experience simple.
To make this project practical, I focused on real user facing flows across eSewa’s public website. I built regression scenarios around those pages that involves a lot of user interaction and heavy navigation.
I started with the landing experience by validating:
homepage load state
visible eSewa branding and logo
hero section visibility
page title correctness
navigation menu presence
I tested the hero section to ensure that key first-screen elements like the headline, CTA button, and banner were visible and correctly rendered.
headline or tagline visibility
call-to-action button presence
hero banner / image rendering
Since mobile adoption is critical for wallet platforms, I verified:
App Store download visibility
Google Play download visibility
To check how easily services can be found, I tested:
navigating to service detail pages or modals
service title visibility
description visibility
back / close interaction availability
I also checked important footer content such as:
contact information
social media links
copyright text
For authentication UX, I tested empty field submission scenarios to verify:
login modal presence
phone number field visibility
password field visibility
sign-in button rendering
validation or error feedback for empty inputs
I tested movement between major sections including:
Services
Merchant / Business
Help / FAQ
About Us
returning back to homepage via logo navigation
Rather than placing every scenario into a single large file, I structured the suite into smaller focused test groups. This made the project easier to maintain and easier to expand later.
esewa-landing.spec.ts → This file includes all the test related to landing page, hero section, footer, download area.
esewa-login.spec.ts → This contains all the checks and teste cases related to empty field validation and authentication UI checks.
esewa-navigation.spec.ts → This covered tests related to Services, Merchant, Help, About, and navigation back to the homepage flows.
Example :
test("Homepage loads correctly", async ({ page }) => {
test.setTimeout(60_000);
await runSteps({
page,
userFlow:
"Validate eSewa homepage loads with key elements and basic interactions",
steps: [
{ description: "Navigate to https://esewa.com.np" },
{ description: "Click on the eSewa logo" },
],
assertions: [
{ assertion: "eSewa logo is visible in the header" },
{ assertion: "User remains on or is redirected to homepage" },
{ assertion: "Top navigation menu with multiple items is visible" },
{ assertion: "Page title contains the word eSewa" },
{ assertion: "Hero section displays a headline or key message" },
{
assertion:
"Clicking the eSewa logo keeps or brings the user to the homepage",
},
{ assertion: "Navigation menu items appear clickable or interactive" },
],
test,
expect,
});
});
All the test suites are available here (I felt it wouldn’t be a good idea to include all the code directly in the blog).
Note: The goal was not to maximize number of tests, but to ensure coverage of meaningful user journeys.
One unexpected issue I faced was related to the OpenRouter API key setup.
As mentioned in the hackathon instructions, the API key is sent automatically to the registered email after signing up for the Passmark hackathon. I ended up registering twice and received two separate emails with different keys.Without realizing it, I used the older API key in my project, which caused authentication errors such as “no user found” during execution.
After some debugging, I revisited my email, identified the correct key, updated the configuration, and the issue was resolved immediately.
This was a simple mistake, but it highlighted how small configuration issues can completely block a working test suite.
After running the test suite on eSewa, the results were not perfectly clean , and that’s where the real insights came from.
Some tests passed fluently without any modification. This was the great motivation to work on this project.
The homepage validation and app download section tests completed successfully, confirming that Passmark handled basic visibility checks and straightforward flows reliably.
These tests involved minimal navigation and fewer steps, which made execution faster and more predictable.
The more interesting behavior appeared in slightly more complex flows.
Several tests failed due to timeouts, including:
hero section content validation
featured service interaction
footer validation
navigation flows like Services, Merchant, and About section.
These weren’t hard failures, they exceeded the default 60 second limit.
This showed that combining multiple steps like scrolling, clicking, and validating content increases execution time significantly in AI driven tests.
I also hit a specific issue on the FAQ page: The page title was visible, but the expected question content was not detected at that moment, causing the assertion to fail.
Two things stood out.
First, navigation heavy tests were slower than expected.
I initially assumed simple UI checks would be trivial, but they required more time because of page transitions and content loading.
Second, how much wording mattered.
Small changes in step descriptions, like being more explicit about what to look for, noticeably improved execution reliability.
After completing and running the test cases, I decided to create a short demo video to push myself further, as I felt a video would be a better way to showcase what Passmark can do.
https://www.youtube.com/watch?v=vixWAl4vjq0
Firstly, I was really surprised by the announcement of this hackathon, and I kept asking myself, "how is it possible to test an application using just plain English?" I was used to the idea that testing requires writing detailed test cases and automating browsers using tools like Selenium and other frameworks.
But when I went through the Passmark documentation and GitHub repository, I was genuinely amazed. I was able to write meaningful test cases simply by describing user actions, without complex setup.
Overall, this was a completely new experience for me. Testing an application using this approach changed how I think about automation.
My POV :
AI won’t replace traditional testing overnight, but tools like Passmark show that the future of testing might be less about writing code, and more about describing intent.