Testing, Tools, and Frameworks Resources

DZone's Featured Testing, Tools, and Frameworks Resources

Mastering Unit Testing and Test-Driven Development in Java

By Maic Moerser

Unit testing is a software testing methodology where individual units or components of software are tested in isolation to check whether it is functioning up to the expectation or not. In Java, it is an essential practice with the help of which an attempt to verify code correctness is made, and an attempt to improve code quality is made. It will basically ensure that the code works fine and the changes are not the point of breakage of existing functionality. Test-Driven Development (TDD) is a test-first approach to software development in short iterations. It is a kind of practice where a test is written before the real source code is written. It pursues the aim of writing code that passes predefined tests and, hence, well-designed, clean, and free from bugs. Key Concepts of Unit Testing Test automation: Use tools for automatic test running, such as JUnit. Asserts: Statements that confirm an expected result within a test. Test coverage: It is the code execution percentage defined by the tests. Test suites: Collection of test cases. Mocks and stubs: Dummy objects that simulate real dependencies. Unit Testing Frameworks in Java: JUnit JUnit is an open-source, simple, and widely used unit testing. JUnit is one of the most popular Java frameworks for unit testing. In other words, it comes with annotations, assertions, and tools required to write and run tests. Core Components of JUnit 1. Annotations JUnit uses annotations to define tests and lifecycle methods. These are some of the key annotations: @Test: Marks a method as a test method. @BeforeEach: Denotes that the annotated method should be executed before each @Test method in the current class. @AfterEach: Denotes that the annotated method should be executed after each @Test method in the current class. @BeforeAll: Denotes that the annotated method should be executed once before any of the @Test methods in the current class. @AfterAll: Denotes that the annotated method should be executed once after all of the @Test methods in the current class. @Disabled: Used to disable a test method or class temporarily. 2. Assertions Assertions are used to test the expected outcomes: assertEquals(expected, actual): Asserts that two values are equal. If they are not, an AssertionError is thrown. assertTrue(boolean condition): Asserts that a condition is true. assertFalse(boolean condition): Asserts that a condition is false. assertNotNull(Object obj): Asserts that an object is not null. assertThrows(Class<T> expectedType, Executable executable): Asserts that the execution of the executable throws an exception of the specified type. 3. Assumptions Assumptions are similar to assertions but used in a different context: assumeTrue(boolean condition): If the condition is false, the test is terminated and considered successful. assumeFalse(boolean condition): The inverse of assumeTrue. 4. Test Lifecycle The lifecycle of a JUnit test runs from initialization to cleanup: @BeforeAll → @BeforeEach → @Test → @AfterEach → @AfterAll This allows for proper setup and teardown operations, ensuring that tests run in a clean state. Example of a Basic JUnit Test Here’s a simple example of a JUnit test class testing a basic calculator: Java import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.AfterEach; import static org.junit.jupiter.api.Assertions.*; class CalculatorTest { private Calculator calculator; @BeforeEach void setUp() { calculator = new Calculator(); } @Test void testAddition() { assertEquals(5, calculator.add(2, 3), "2 + 3 should equal 5"); } @Test void testMultiplication() { assertAll( () -> assertEquals(6, calculator.multiply(2, 3), "2 * 3 should equal 6"), () -> assertEquals(0, calculator.multiply(0, 5), "0 * 5 should equal 0") ); } @AfterEach void tearDown() { // Clean up resources, if necessary calculator = null; } } Dynamic Tests in JUnit 5 JUnit 5 introduced a powerful feature called dynamic tests. Unlike static tests, which are defined at compile-time using the @Test annotation, dynamic tests are created at runtime. This allows for more flexibility and dynamism in test creation. Why Use Dynamic Tests? Parameterized testing: This allows you to create a set of tests that execute the same code but with different parameters. Dynamic data sources: Create tests based on data that may not be available at compile-time (e.g., data from external sources). Adaptive testing: Tests can be generated based on the environment or system conditions. Creating Dynamic Tests JUnit provides the DynamicTest class for creating dynamic tests. You also need to use the @TestFactory annotation to mark the method that returns the dynamic tests. Example of Dynamic Tests Java import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import java.util.Arrays; import java.util.Collection; import java.util.stream.Stream; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.DynamicTest.dynamicTest; class DynamicTestsExample { @TestFactory Stream<DynamicTest> dynamicTestsFromStream() { return Stream.of("apple", "banana", "lemon") .map(fruit -> dynamicTest("Test for " + fruit, () -> { assertEquals(5, fruit.length()); })); } @TestFactory Collection<DynamicTest> dynamicTestsFromCollection() { return Arrays.asList( dynamicTest("Positive Test", () -> assertEquals(2, 1 + 1)), dynamicTest("Negative Test", () -> assertEquals(-2, -1 + -1)) ); } } Creating Parameterized Tests In JUnit 5, you can create parameterized tests using the @ParameterizedTest annotation. You'll need to use a specific source annotation to supply the parameters. Here's an overview of the commonly used sources: @ValueSource: Supplies a single array of literal values. @CsvSource: Supplies data in CSV format. @MethodSource: Supplies data from a factory method. @EnumSource: Supplies data from an Enum. Example of Parameterized Tests Using @ValueSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; import static org.junit.jupiter.api.Assertions.assertTrue; class ValueSourceTest { @ParameterizedTest @ValueSource(strings = {"apple", "banana", "orange"}) void testWithValueSource(String fruit) { assertTrue(fruit.length() > 4); } } Using @CsvSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import static org.junit.jupiter.api.Assertions.assertEquals; class CsvSourceTest { @ParameterizedTest @CsvSource({ "test,4", "hello,5", "JUnit,5" }) void testWithCsvSource(String word, int expectedLength) { assertEquals(expectedLength, word.length()); } } Using @MethodSource Java import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.MethodSource; import java.util.stream.Stream; import static org.junit.jupiter.api.Assertions.assertTrue; class MethodSourceTest { @ParameterizedTest @MethodSource("stringProvider") void testWithMethodSource(String word) { assertTrue(word.length() > 4); } static Stream<String> stringProvider() { return Stream.of("apple", "banana", "orange"); } } Best Practices for Parameterized Tests Use descriptive test names: Leverage @DisplayName for clarity. Limit parameter count: Keep the number of parameters manageable to ensure readability. Reuse methods for data providers: For @MethodSource, use static methods that provide the data sets. Combine data sources: Use multiple source annotations for comprehensive test coverage. Tagging in JUnit 5 The other salient feature in JUnit 5 is tagging: it allows for assigning their own custom tags to tests. Tags allow, therefore, a way to group tests and later execute groups selectively by their tag. This would be very useful for managing large test suites. Key Features of Tagging Flexible grouping: Multiple tags can be applied to a single test method or class, so flexible grouping strategies can be defined. Selective execution: Sometimes it may be required to execute only the desired group of tests by adding tags. Improved organization: Provides an organized way to set up tests for improved clarity and maintainability. Using Tags in JUnit 5 To use tags, you annotate your test methods or test classes with the @Tag annotation, followed by a string representing the tag name. Example Usage of @Tag Java import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; @Tag("fast") class FastTests { @Test @Tag("unit") void fastUnitTest() { // Test logic for a fast unit test } @Test void fastIntegrationTest() { // Test logic for a fast integration test } } @Tag("slow") class SlowTests { @Test @Tag("integration") void slowIntegrationTest() { // Test logic for a slow integration test } } Running Tagged Tests You can run tests with specific tags using: Command line: Run the tests by passing the -t (or --tags) argument to specify which tags to include or exclude.mvn test -Dgroups="fast" IDE: Most modern IDEs like IntelliJ IDEA and Eclipse allow selecting specific tags through their graphical user interfaces. Build tools: Maven and Gradle support specifying tags to include or exclude during the build and test phases. Best Practices for Tagging Consistent tag names: Use a consistent naming convention across your test suite for tags, such as "unit", "integration", or "slow". Layered tagging: Apply broader tags at the class level (e.g., "integration") and more specific tags at the method level (e.g., "slow"). Avoid over-tagging: Do not add too many tags to a single test, which can reduce clarity and effectiveness. JUnit 5 Extensions The JUnit 5 extension model allows developers to extend and otherwise customize test behavior. They provide a mechanism for extending tests with additional functionality, modifying the test execution lifecycle, and adding new features to your tests. Key Features of JUnit 5 Extensions Customization: Modify the behavior of test execution or lifecycle methods. Reusability: Create reusable components that can be applied to different tests or projects. Integration: Integrate with other frameworks or external systems to add functionality like logging, database initialization, etc. Types of Extensions Test Lifecycle Callbacks BeforeAllCallback, BeforeEachCallback, AfterAllCallback, AfterEachCallback. Allow custom actions before and after test methods or test classes. Parameter Resolvers ParameterResolver. Inject custom parameters into test methods, such as mock objects, database connections, etc. Test Execution Condition ExecutionCondition. Enable or disable tests based on custom conditions (e.g., environment variables, OS type). Exception Handlers TestExecutionExceptionHandler. Handle exceptions thrown during test execution. Others TestInstancePostProcessor, TestTemplateInvocationContextProvider, etc. Customize test instance creation, template invocation, etc. Implementing Custom Extensions To create a custom extension, you need to implement one or more of the above interfaces and annotate the class with @ExtendWith. Example: Custom Parameter Resolver A simple parameter resolver that injects a string into the test method: Java import org.junit.jupiter.api.extension.*; public class CustomParameterResolver implements ParameterResolver { @Override public boolean supportsParameter(ParameterContext parameterContext, ExtensionContext extensionContext) { return parameterContext.getParameter().getType().equals(String.class); } @Override public Object resolveParameter(ParameterContext parameterContext, ExtensionContext extensionContext) { return "Injected String"; } } Using the Custom Extension in Tests Java import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @ExtendWith(CustomParameterResolver.class) class CustomParameterTest { @Test void testWithCustomParameter(String injectedString) { System.out.println(injectedString); // Output: Injected String } } Best Practices for Extensions Separation of concerns: Extensions should have a single, well-defined responsibility. Reusability: Design extensions to be reusable across different projects. Documentation: Document how the extension works and its intended use cases. Unit testing and Test-Driven Development (TDD) offer significant benefits that positively impact software development processes and outcomes. Benefits of Unit Testing Improved Code Quality Detection of bugs: Unit tests detect bugs early in the development cycle, making them easier and cheaper to fix. Code integrity: Tests verify that code changes don't break existing functionality, ensuring continuous code integrity. Simplifies Refactoring Tests serve as a safety net during code refactoring. If all tests pass after refactoring, developers can be confident that the refactoring did not break existing functionality. Documentation Tests serve as live documentation that illustrates how the code is supposed to be used. They provide examples of the intended behavior of methods, which can be especially useful for new team members. Modularity and Reusability Writing testable code encourages modular design. Code that is easily testable is generally also more reusable and easier to understand. Reduces Fear of Changes A comprehensive test suite helps developers make changes confidently, knowing they will be notified if anything breaks. Regression Testing Unit tests can catch regressions, where previously working code stops functioning correctly due to new changes. Encourages Best Practices Developers tend to write cleaner, well-structured, and decoupled code when unit tests are a priority. Benefits of Test-Driven Development (TDD) Ensures test coverage: TDD ensures that every line of production code is covered by at least one test. This provides comprehensive coverage and verification. Focus on requirements: Writing tests before writing code forces developers to think critically about requirements and expected behavior before implementation. Improved design: The incremental approach of TDD often leads to better system design. Code is written with testing in mind, resulting in loosely coupled and modular systems. Reduces debugging time: Since tests are written before the code, bugs are caught early in the development cycle, reducing the amount of time spent debugging. Simplifies maintenance: Well-tested code is easier to maintain because the tests provide instant feedback when changes are introduced. Boosts developer confidence: Developers are more confident in their changes knowing that tests have already validated the behavior of their code. Facilitates collaboration: A comprehensive test suite enables multiple developers to work on the same codebase, reducing integration issues and conflicts. Helps identify edge cases: Thinking through edge cases while writing tests helps to identify unusual conditions that could be overlooked otherwise. Reduces overall development time: Although TDD may initially seem to slow development due to the time spent writing tests, it often reduces the total development time by preventing bugs and reducing the time spent on debugging and refactoring. Conclusion By leveraging unit testing and TDD in Java with JUnit, developers can produce high-quality software that's easier to maintain and extend over time. These practices are essential for any professional software development workflow, fostering confidence and stability in your application's codebase. More

Queuing Theory for Non-Functional Testing

By Stelios Manioudakis, PhD

CORE

Queuing theory is a branch of mathematics that analyzes how waiting lines (queues) form and behave in systems. In the context of non-functional software testing, it provides a valuable tool for understanding how a system performs under varying loads. By analyzing queue lengths, waiting times, and server utilization, queuing models can help predict potential bottlenecks and performance issues before they occur in real-world use. This article starts with the basics of queuing theory for non-functional software testing. Benefits and limitations will be addressed. As a case study, we will explore a sample of queuing models that could be appropriate for mobile gaming applications. Finally, a set of tools that can be used will be explored with their pros and cons. Key Concepts in Queuing Theory Queuing theory provides mathematical models that can be used for non-functional testing. First, we will explain basic queuing concepts to understand how to use them. Arrival rate (λ): This refers to the average number of tasks or requests entering the system per unit of time. For example, it could represent the number of customers arriving at a bank per minute or the number of network packets arriving at a router per second. Service time (μ): This represents the average time it takes for a resource to complete a task. In a bank, it might be the average time a teller spends with a customer. In a network, it could be the average time it takes to process a data packet. Queue length (L): This refers to the number of tasks waiting for service at any given time. It is directly related to the arrival rate, service time, and the number of available resources. Number of servers (S): This refers to the resources available to handle tasks. In a bank, it's the number of tellers available to serve customers. In a network, it could be the number of processing cores in a server or the number of available network channels. Queue discipline: This defines how tasks are selected from the queue for service. Some common disciplines include: First-In-First-Out (FIFO): The first task to enter the queue is the first to be served. This is often used in situations like checkout lines or waiting rooms. Priority queue: Certain tasks are assigned a higher priority and are served before lower-priority tasks, even if they arrive earlier. This is used in situations where certain tasks are critical and require immediate attention. Shortest Processing Time (SPT): The task with the shortest expected service time is served first. This can be beneficial for minimizing average waiting times overall. Applications in Non-Functional Testing A sample of applications in non-functional software testing includes the following: Load Testing By simulating realistic user loads with specific arrival rates and service times based on queuing models, load testing tools can evaluate system performance under stress. This helps identify potential bottlenecks like overloaded servers, slow database queries, or inefficient code execution. By analyzing queue lengths and waiting times during the load test, you can pinpoint areas where the system struggles and implement improvements before deployment. Capacity Planning Queuing theory models can be integrated with testing tools to determine the system's breaking point or optimal resource allocation. The model can predict how the system behaves with different numbers of servers, allowing you to find the sweet spot between adequate performance and cost-effective resource utilization. This helps ensure the system can handle expected user traffic without compromising performance or incurring unnecessary costs for over-provisioning. Performance Benchmarking Queuing models can be used to compare the performance of different system configurations or architectures. By simulating the same workload on different system setups, you can evaluate which configuration delivers the best performance in terms of waiting times and server utilization. This can be particularly helpful when choosing between different hardware or software options. Other Useful Concepts in Queuing Theory Little's Law This fundamental relationship in queuing theory states that the average number of tasks in the system (L) is equal to the average arrival rate (λ) multiplied by the average waiting time (W). This allows you to estimate one of these values if you know the other two. Kendall-Lee Notation This notation is a standardized way to describe queuing systems based on arrival distribution, service distribution, number of servers, and queue discipline. Understanding this notation helps classify different queuing models and choose the appropriate one for analysis. Open vs. Closed Queuing Systems Open queuing systems allow tasks to enter and leave the system. Closed queuing systems have a fixed number of tasks circulating within the system. Choosing the right model depends on the system being analyzed. Limitations of Using Queuing Theory As with all theories, queuing theory is based on assumptions. The benefits that we can enjoy by employing queuing theory in non-functional testing are largely affected by how realistic are those assumptions. Simplified Assumptions Queuing models often rely on simplifying assumptions to make the math tractable. These assumptions include: Stable arrival rates and service times: Real-world systems might experience fluctuations in arrival rates and service times. Queuing models might not accurately reflect such dynamic behavior. Infinite queues: In reality, queues might have a finite capacity. If the queue becomes full, new arrivals might be rejected, leading to system instability. Queuing models with finite queues can be more complex but offer a more realistic representation. The Case of Mobile Gaming Mobile games, especially those with online multiplayer components or microtransaction systems, often involve interactions that we can model using queuing theory. We will analyze a list of possible queuing models appropriate for mobile games. The list is not exhaustive but it can explain the reasoning for using different models and their benefits. 1. M/M/1 Queuing System With Network Latency In mobile games with online multiplayer features, players may connect to a central server to interact with each other. This scenario can be modeled as an M/M/1 queuing system, where players are the arriving entities, and the server acts as the single server. Incorporating network latency into the model allows developers to analyze the impact of delays on player experience and design strategies to mitigate them. Understanding the queuing behavior helps optimize server capacity and network infrastructure to minimize latency and enhance the gameplay experience. 2. M/G/1 Queuing System for In-Game Purchases Mobile games often include in-game stores where players can make purchases using real or virtual currency. The arrival of purchase requests and the service times for processing these transactions may not follow exponential distributions typical of M/M/1 systems. An M/G/1 queuing system, where the service time distribution is generalized, can be more appropriate for modeling in-game purchase transactions. Analyzing this model helps game developers optimize the payment processing system, streamline transaction flows, and manage server resources efficiently. 3. Finite Source Queuing Models for Limited Resources Many mobile games have limited resources, such as virtual items, game levels, or server capacity. Players may need to wait in a queue to access these resources, especially during peak usage periods. Finite source queuing models, such as the M/M/c/K model (with c servers and a finite queue of size K), are suitable for analyzing scenarios where there are constraints on the availability of resources. By understanding queue dynamics and resource utilization, developers can implement strategies to balance resource allocation, reduce wait times, and optimize player satisfaction. 4. Dynamic Queueing Models for Matchmaking Matchmaking algorithms are crucial for ensuring balanced and enjoyable gameplay experiences in multiplayer mobile games. These algorithms often involve queuing mechanisms to match players with similar skill levels or preferences. Dynamic queueing models, such as the M/M/1/K queue with dynamic arrival rates or the Erlang queuing model with variable service rates, can be employed to optimize matchmaking systems. By adjusting queue parameters dynamically based on player behavior, game developers can achieve faster and fairer matchmaking results. This may lead to increased player engagement and retention. Queuing Theory Saves the Launch Day A mobile game development company was gearing up for the highly anticipated launch of their latest game. Based on pre-registration numbers and social media buzz, they were expecting a massive influx of players on launch day. Their concern was twofold: ensuring smooth gameplay for all users and avoiding server crashes due to overload. The development team decided to use queuing theory to create a model of their game server infrastructure. Model Derivation They identified the game server system as an M/M/c queuing system, meaning: M: Player arrivals follow a Poisson distribution (random and independent). M: The time it takes to process a player's request (e.g., joining a game, updating the game state) follows a Poisson distribution (random and independent). c: Represents the number of available game servers (acting as multiple queues) Key Performance Metrics Using queuing theory formulas, they calculated the following metrics: Arrival rate (λ): Estimated based on pre-registration data and industry benchmarks for similar game launches Service time (μ): Measured by analyzing the average time it takes to process player requests during internal testing Server utilization (ρ): ρ = λ / (c * μ). This metric indicates how busy each server is on average. Model Analysis The key aspect of the model was understanding how server utilization (ρ) would change with different server configurations (number of servers, "c"). High server utilization (ρ > 0.8): Signifies servers are overloaded, leading to potential queueing delays, slow gameplay, and increased risk of crashes Low server utilization (ρ < 0.5): Indicates underutilized servers, which might be cost-inefficient, but ensures smooth gameplay Taking Action Using the queuing model, the team conducted a series of tests: Scenario 1: Existing Server Configuration The model predicted a server utilization exceeding 80% during peak launch hours, potentially leading to performance issues and frustrated players. Scenario 2: Adding 20% More Servers The model showed a utilization dropping to around 65%, offering a significant performance improvement while maintaining some buffer for unexpected player surges. Scenario 3: Doubling the Servers Utilization dropped further to around 40%, but the additional server cost might not be justified if player growth was slower than anticipated. The Decision Based on the model's predictions, the team decided to add 20% more servers to their existing infrastructure. This provided a significant performance boost without incurring excessive costs. Additionally, they implemented auto-scaling rules to automatically provision additional servers if player traffic exceeded pre-defined thresholds. The Outcome Launch day arrived, and the company saw a record number of players joining the new game. However, thanks to the queuing model and proactive server scaling, the servers handled the load efficiently. Players experienced smooth gameplay without major delays or crashes. Tool Selection There are various testing tools available that incorporate queuing theory principles. Choosing the right tool depends on the complexity of the system, the desired level of detail, and the specific testing goals. The following list is by no means exhaustive. Microsoft Excel with queuing model add-ins: Pros: Free, readily available for most users, easy to learn basic formulas Cons: Limited functionality, error-prone with complex models, not ideal for large-scale testing Online queuing model calculators: Pros: Free, user-friendly interface, good for quick estimations Cons: Limited model options, may not capture specific system details, limited customization JMeter: Pros: Open-source, robust load testing capabilities, supports basic queuing theory integration for user load simulation Cons: Setting up queuing models can be complex, requires scripting knowledge for advanced features Apache JMeter plugin - queueing theory: Pros: Extends JMeter functionalities with queuing theory models, allows for analyzing server utilization and wait times Cons: Relies on JMeter's learning curve, additional configuration needed for queuing features AppDynamics Pros: Commercial tool with a good user interface, offers performance monitoring with queuing theory insights (queue lengths, wait times) Cons: Subscription-based cost, may require training for advanced features AnyLogic: Pros: Powerful simulation software, integrates with queuing models to create complex scenarios, provides detailed performance reports Cons: Steeper learning curve, requires modeling expertise, significant cost for commercial licenses Wrapping Up Queuing theory is a valuable option for optimizing performance and resource allocation in various software development scenarios. By understanding the core queuing models and their limitations, development teams can leverage testing tools. They can analyze server utilization, identify potential bottlenecks, and make data-driven decisions. Our task could be to ensure smooth gameplay for a mobile game launch, to optimize infrastructure for a rapidly growing company, or to simply choose the best cloud platform for an application. In any case, queuing theory may empower developers to navigate the complexities of system load and create a seamless user experience. More

Wireshark and tcpdump: A Debugging Power Couple

By Shai Almog

CORE

Comprehensive Guide to Unit Testing Spring AOP Aspects

By Reza Ganji

CORE

Reliability Models and Metrics for Test Engineering

By Stelios Manioudakis, PhD

CORE

Software Testing as a Debugging Tool

Debugging is not just about identifying errors — it's about instituting a reliable process for ensuring software health and longevity. In this post, we discuss the role of software testing in debugging, including foundational concepts and how they converge to improve software quality. As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book. The Intersection of Debugging and Testing Debugging and testing play distinct roles in software development. Debugging is the targeted process of identifying and fixing known bugs. Testing, on the other hand, encompasses an adjacent scope, identifying unknown issues by validating expected software behavior across a variety of scenarios. Both are a part of the debug fix cycle which is a core concept in debugging. Before we cover the cycle we should first make sure we're aligned on the basic terminology. Unit Tests Unit tests are tightly linked to debugging efforts, focusing on isolated parts of the application—typically individual functions or methods. Their purpose is to validate that each unit operates correctly in isolation, making them a swift and efficient tool in the debugging arsenal. These tests are characterized by their speed and consistency, enabling developers to run them frequently, sometimes even automatically as code is written within the IDE. Since software is so tightly bound it is nearly impossible to compose unit tests without extensive mocking. Mocking involves substituting a genuine component with a stand-in that returns predefined results, thus a test method can simulate scenarios without relying on the actual object. This is a powerful yet controversial tool. By using mocking we're in effect creating a synthetic environment that might misrepresent the real world. We're reducing the scope of the test which might perpetuate some bugs. Integration Tests Opposite to unit tests, integration tests examine the interactions between multiple units, providing a more comprehensive picture of the system's health. While they cover broader scenarios, their setup can be more complex due to the interactions involved. However, they are crucial in catching bugs that arise from the interplay between different software components. In general, mocking can be used in integration tests but it is discouraged. They take longer to run and are sometimes harder to set up. However, many developers (myself included) would argue that they are the only benchmark for quality. Most bugs express themselves in the seams between the modules and integration tests are better at detecting that. Since they are far more important some developers would argue that unit tests are unnecessary. This isn't true, unit test failures are much easier to read and understand. Since they are faster we can run them during development, even while typing. In that sense, the balance between the two approaches is the important part. Coverage Coverage is a metric that helps quantify the effectiveness of testing by indicating the proportion of code exercised by tests. It helps identify potential areas of the code that have not been tested, which could harbor undetected bugs. However, striving for 100% coverage can be a case of diminishing returns; the focus should remain on the quality and relevance of the tests rather than the metric itself. In my experience, chasing high coverage numbers often results in bad test practices that persist in problems. It is my opinion that unit tests should be excluded from coverage metrics due to the importance of integration tests to overall quality. To get a sense of quality coverage should focus on integration and end-to-end tests. The Debug-Fix Cycle The debug-fix cycle is a structured approach that integrates testing into the debugging process. The stages include identifying the bug, creating a test that reproduces the bug, fixing the bug, verifying the fix with the test, and finally, running the application to ensure the fix works in the live environment. This cycle emphasizes the importance of testing in not only identifying but also in preventing the recurrence of bugs. Notice that this is a simplified version of the cycle with a focus on the testing aspect only. The full cycle includes a discussion of the issue tracking and versioning as part of the whole process. I discuss this more in-depth in other posts in the series and my book. Composing Tests With Debuggers A powerful feature of using debuggers in test composition is their ability to "jump to line" or "set value." Developers can effectively reset the execution to a point before the test and rerun it with different conditions, without recompiling or rerunning the entire suite. This iterative process is invaluable for achieving desired test constraints and improves the quality of unit tests by refining the input parameters and expected outcomes. Increasing test coverage is about more than hitting a percentage; it's about ensuring that tests are meaningful and that they contribute to software quality. A debugger can significantly assist in this by identifying untested paths. When a test coverage tool highlights lines or conditions not reached by current tests, the debugger can be used to force execution down those paths. This helps in crafting additional tests that cover missed scenarios, ensuring that the coverage metric is not just a number but a true reflection of the software's tested state. In this case, you will notice that the next line in the body is a rejectValue call which will throw an exception. I don’t want an exception thrown as I still want to test all the permutations of the method. I can drag the execution pointer (arrow on the left) and place it back at the start of the method. Test-Driven Development How does all of this fit with disciplines like Test-Driven Development (TDD)? It doesn't fit well. Before we get into that let's revisit the basics of TDD. Weak TDD typically means just writing tests before writing the code. Strong TDD involves a red-green-refactor cycle: Red: Write a test that fails because the feature it tests isn't implemented yet. Green: Write the minimum amount of code necessary to make the test pass. Refactor: Clean up the code while ensuring that tests continue to pass. This rigorous cycle guarantees that new code is continually tested and refactored, reducing the likelihood of complex bugs. It also means that when bugs do appear, they are often easier to isolate and fix due to the modular and well-tested nature of the codebase. At least, that's the theory. TDD can be especially advantageous for scripting and loosely typed languages. In environments lacking the rigid structure of compilers and linters, TDD steps in to provide the necessary checks that would otherwise be performed during compilation in statically typed languages. It becomes a crucial substitute for compiler/linter checks, ensuring that type and logic errors are caught early. In real-world application development, TDD's utility is nuanced. While it encourages thorough testing and upfront design, it can sometimes hinder the natural flow of development, especially in complex systems that evolve through numerous iterations. The requirement for 100% test coverage can lead to an unnecessary focus on fulfilling metrics rather than writing meaningful tests. The biggest problem in TDD is its focus on unit testing. TDD is impractical with integration tests as the process would take too long. But as we determined at the start of this post, integration tests are the true benchmark for quality. In that test TDD is a methodology that provides great quality for arbitrary tests, but not necessarily great quality for the final product. You might have the best cog in the world, but if doesn't fit well into the machine then it isn't great. Final Word Debugging is a tool that not only fixes bugs but also actively aids in crafting tests that bolster software quality. By utilizing debuggers in test composition and increasing coverage, developers can create a suite of tests that not only identifies existing issues but also guards against future ones, thus ensuring the delivery of reliable, high-quality software. Debugging lets us increase coverage and verify edge cases effectively. It's part of a standardized process for issue resolution that's critical for reliability and prevents regressions.

By Shai Almog

CORE

If Software Quality Is Everybody’s Responsibility, So Is Failure

In many large organizations, software quality is primarily viewed as the responsibility of the testing team. When bugs slip through to production, or products fail to meet customer expectations, testers are the ones blamed. However, taking a closer look, quality — and likewise, failure — extends well beyond any one discipline. Quality is a responsibility shared across an organization. When quality issues arise, the root cause is rarely something testing alone could have prevented. Typically, there were breakdowns in communication, unrealistic deadlines, inadequate design specifications, insufficient training, or corporate governance policies that incentivized rushing. In other words, quality failures tend to stem from broader organizational and leadership failures. Scapegoating testers for systemic issues is counterproductive. It obscures the real problems and stands in the way of meaningful solutions to quality failings. Testing in Isolation In practice, all too often, testing teams still work in isolation from the rest of the product development lifecycle. They are brought in at the end, given limited information, and asked to validate someone else’s work. Under these conditions, their ability to prevent defects is severely constrained. For example, without access to product requirement documents, test cases may overlook critical functions that need validation. With short testing timelines, extensive test coverage further becomes impossible. Without insight into design decisions or access to developers, some defects found in testing prove impossible to diagnose effectively. Testers are often parachuted in when the time and cost of repairing a defect has grown to be unfeasible. In this isolated model, testing serves as little more than a final safety check before release. The burden of quality is passed almost entirely to the testers. When the inevitable bugs still slip through, testers then make for easy scapegoats. Who Owns Software Quality? In truth, responsibility for product quality is distributed across an organization. So, what can you do? Quality is everyone’s responsibility. Image sources: Kharnagy (Wikipedia), under CC BY-SA 4.0 license, combined with an image from Pixabay. Executives and leadership teams — Set the tone and policies around quality, balancing it appropriately against other priorities like cost and schedule. Meanwhile, provide the staffing, resources, and timescale needed for a mature testing effort. Product Managers — Gather user requirements, define expected functionality, and support test planning. Developers — Follow secure coding practices, perform unit testing, enable automated testing, and respond to defects uncovered in testing. User experience designers — Consider quality and testability during UX design. Conduct user acceptance testing on prototypes. Information security — Perform security reviews of code, architectures, and configurations. Guide testing-relevant security use cases. Testers — Develop test cases based on user stories, execute testing, log defects, perform regression test fixes, and report on quality to stakeholders. Operations — Monitor systems once deployed, gather production issues, and report data to inform future testing. Customers — Voice your true quality expectations, participate in UAT, and report real-world issues once launched. As this illustrates, no one functional area owns quality alone. Testers contribute essential verification, but quality is truly everyone’s responsibility. Governance Breakdowns Lead to Quality Failures In a 2023 episode of the "Why Didn’t You Test That?" podcast, Marcus Merrell, Huw Price, and I discussed how testing remains treated as a “janitorial” effort and cost center, and how you can align testing and quality. When organizations fail to acknowledge the shared ownership of software quality, governance issues arise that enable quality failures: Unrealistic deadlines — Attempting to achieve overly aggressive schedules often comes at the expense of quality and sufficient testing timelines. Leadership teams must balance market demands against release readiness. Insufficient investment — Success requires appropriate staffing and support for all areas that influence quality. These range from design and development to development to testing. Underinvestment leads to unhealthy tradeoffs. Lack of collaboration — Cross-functional coordination produces better quality than work done in silos. Governance policies should foster collaboration across product teams, not hinder it. Misaligned priorities — Leadership should incentivize balanced delivery, not just speed or cost savings. Quality cannot be someone else’s problem. Lack of transparency — Progress reporting should incorporate real metrics on quality. Burying or obscuring defects undermines governance. Absence of risk management — Identifying and mitigating quality risks through appropriate action requires focus from project leadership. Lacking transparency about risk prevents proper governance. When these governance breakdowns occur, quality suffers, and failures follow. However, the root causes trace back to organizational leadership and culture, not solely the testing function. The Costs of Obscuring Systemic Issues Blaming testers for failures caused by systemic organizational issues leads to significant costs: Loss of trust — When testers become scapegoats, it erodes credibility and trust in the testing function, inhibiting their ability to advocate for product quality. Staff turnover — Testing teams experience higher turnover when the broader organization fails to recognize their contributions and value. Less collaboration — Other groups avoid collaborating with testers perceived as bottlenecks or impediments rather than partners. Reinventing the wheel — Lessons from past governance breakdowns go unlearned, leading those issues to resurface in new forms down the line. Poorer customer experiences — Ultimately, obscuring governance issues around quality leads to more negative customer experiences that damage an organization’s reputation and bottom line. Taking Ownership of Software Quality Elevating quality as an organization-wide responsibility is essential for governance, transparency, and risk management. Quality cannot be the burden of one isolated function, and leadership should foster a culture that values quality intrinsically, rather than viewing it as an afterthought or checkbox. To build ownership, organizations need to shift testing upstream, integrating it earlier into requirements planning, design reviews, and development processes. It also requires modernizing the testing practice itself, utilizing the full range of innovation available: from test automation, shift-left testing, and service virtualization, to risk-based test case generation, modeling, and generative AI. With a shared understanding of who owns quality, governance policies can better balance competing demands around cost, schedule, capabilities, and release readiness. Testing insights will inform smarter tradeoffs, avoiding quality failures and the finger-pointing that today follows them. This future state reduces the likelihood of failures — but also acknowledges that some failures will still occur despite best efforts. In these cases, organizations must have a governance model to transparently identify root causes across teams, learn from them, and prevent recurrence. In a culture that values quality intrinsically, software testers earn their place as trusted advisors, rather than get relegated to fault-finders. They can provide oversight and validation of other teams’ work without fear of backlash. And their expertise will strengthen rather than threaten collaborative delivery. With shared ownership, quality ceases to be a “tester problem” at all. It becomes an organizational value that earns buy-in across functional areas. Leadership sets the tone for an understanding that if quality is everyone’s responsibility — so too is failure.

By Rich Jordan

Automation Testing in CI/CD Pipelines: Types and Stages

In a previous article, we discussed the various use cases wherein agile teams would automate their test cases. One of the cases was when teams wanted to integrate tests with every build and have a continuous integration as a part of the build process. In this article, we will address integrating tests in a continuous integration/continuous delivery platform. Let’s start with the basics first. What Is Automation Testing? Software testing refers to the execution of tests according to a set of test cases and comparing the results of actual and predicted outcomes. Several steps and testing techniques are followed in the process. Testing is essential to ensuring the quality of the product. This process is typically done manually by a team of testers. However, in automation testing, the process is automated using software tools and technology. Here, rather than resorting to a manual effort, scripts are created and test cases are run automatically. Automation testing eradicates many problems such as human error, coverage area, etc. It also saves time and enhances the ease of conducting tests with increased efficiency and effectiveness. Automation Testing in CI/CD Pipelines Automation testing is a core part of CI/CD pipelines because tests that run fast can provide early feedback to the developer. A problem or a bug that is identified early has the possibility to be rectified early. Therefore, the released product will be more accurate and bug-free. This enhances the overall quality of the product, which garners customer satisfaction. To put it succinctly, the advantages of automation testing in the CI/CD pipeline are as follows: Aligns with the CI/CD idea of "build fast, fail fast" Reduces manual effort, which saves time and the possibility of error. Test results are more accurate with the increased number of test cases and can and cover a wider area. Get immediate feedback on any problem Multiple test results can be generated and compared to ensure quality and consistency Types of Automation Testing Used in CI/CD Pipelines Unit testing: It is a low-level test undertaken after a module has been coded and reviewed. Test cases are designed to test individual components. The purpose is to make sure each component works the way it is supposed to under any circumstance. Integration testing: Integration refers to testing the interaction of components within the application. This testing is carried out after all the modules have been unit tested. The primary objective is to test the module interfaces and check that there are no errors in parameter passing when one module invokes the functionality of another module. System testing: Tests are designed to validate a fully-developed system and make sure it adheres to the requirement specification document. Typically, at this stage, software is ready for use by potential users. These kinds of system tests are called alpha (carried out by a team of developers within the organization), beta (performed by a selected group of users/customers), or acceptance testing (performed by users/customers to determine acceptance of delivery of the system). What Are Continuous Integration and Continuous Delivery? In simple terms, continuous integration allows development teams to integrate their code into a shared repository. This helps maintain code quality and identify potential issues with the local version of the code at an early stage. Continuous delivery is often called "Continuous Deployment" as well. Everything that is continuously merged by the development team is continuously deployed to the live environment. Since most developers work in parallel, continuously integrating their code into one repository would mean that the master branch is continuously updated with new features. To ensure that there is no compromise in the code quality with so many changes happening rapidly, testing must happen at the same pace. It should be no surprise that manual testing in this environment would not be the best approach to achieve this. Automated testing is the key to successful testing in a CI/CD pipeline. 9 Continuous Delivery Stages Develop: The developer builds the code according to the project requirements or the feature requests. Writing tests: Once the code is written, tests need to be written. At this point, these tests are usually unit tests written by the developers. Local Testing: This is then locally tested to check whether all the tests pass and to ensure the code does not break. Often, a percentage is set as the pass rate that the tests running need to meet. Rebase and Resolve conflict: In an actual development scenario, there will be multiple people merging their code. Developers need to make sure that their branch is updated at all times. Updating the branch with the latest merged code is called "rebasing." Once it’s rebased, there will likely be some conflicts that need to be resolved. After that, the tests are run again against the rebased code. Commit: Once the tests have passed, the code is then ready to be committed with all the changes. Build: The source code developed is then combined to build a deployment artifact that can be run on an instance, like the server if the environment is on-premises. This code is now ready to be deployed to different testing environments. UAT: The code is then deployed to a test server where testers start to test the feature. These tests can be automated as well as manual. Merge: If the commit that’s under testing is approved by the testers, this is then merged into the master branch. Production deployment: Once the code is merged, it is then deployed to production. The above process needs to be done with every build coded by the developers. Where Does Automation Testing Fall in This CI/CD Pipeline? Automated testing ideally happens once the build stage has been completed and the code can be deployed. Unit tests, UI tests, and integration tests can all be run at this stage. These tests help in ensuring that the code meets a standard of quality. This phase can last from a few minutes to a couple of hours depending on how the automation is architected. Tests can be run in parallel to execute them more quickly. If a code fails during the test phase, the build can be rejected without further investing in any manual testing time. Tools Used for CI/CD Jenkins: Jenkins is an open-source tool that is used for continuous integration. It’s free to use and jobs can be configured both by the interface as well as scripts. Travis CI: This tool is free of cost for open-source projects, hosted by GitHub. Gitlab: Gitlab is a version control tool that has its own cloud-based CI methodology. It is supported on multiple platforms which have both free and paid versions. Bamboo: Bamboo is a CI tool by Jira. If your organization uses Jira, then it would be beneficial to check this tool out. It supports automated merging on approval of tickets as well. Best Practices for CI/CD Pipeline to Make the Best Out of Test Automation Incremental changes: It is always advisable to follow a feature-by-feature approach. If the feature is really big, it is good to break it down into smaller and quicker-to-test features. This is important in terms of automation because if there is an issue, it is easier to figure out the root cause. If your commit is too big, isolating the cause of an issue would be a tough task. Identify what can be automated: It is very common for teams to dive fast and say, "Let’s automate everything," but this is a common mistake. We must know the purpose of automation and identify the test cases that should be automated. Parallel Tests: Tests should be run in parallel to make testing more efficient and timely. It can greatly reduce the time taken to run tests and thus give the results much faster. But it’s not sufficient to just execute these tests in parallel; it is also important to scale the server size where the tests are running in order to actually make them faster. Conclusion Automating tests is an important part of the successful deployment of projects while maintaining a standard of quality. Ensuring tests are run at every stage gives good transparency on the quality of the code. Bugs can be discovered at an early stage, and any delays that might be caused by them can be addressed promptly. Having a CI/CD pipeline with integrated tests helps in speeding up the testing and deployment process.

By Arjun Sharma

How To Share Variables Between Tests in Cypress

Best Practices for Sharing Variables in Cypress Tests Cypress issues a command in a series. Each link in the chain is connected to the one before it as well as the one after. Cypress will automatically wait for the previous command to finish, ensuring that you don’t encounter race situations. I’ll use one as an example: JavaScript cy .get('li') .should('have.length', 5) // ensure the previous command retrieves elements .eq(4) // ensure the previous assertion is successful .click() // ensure the previous command is complete Again, no command will be executed until the previous one has ended. The test fails (often in 4 seconds) if any of the commands are not completed in a timely manner. Let’s take a second to consider the scenario from a different angle. JavaScript it('assigns value to variable', () => { // Executed instantly, outside of chain console.log('>>> log one') let boardId cy.request('/api/boards') .then( response => { console.log('>>> log two') boardId = response.body[0].id }) // Executed instantly, outside of chain console.log('>>> log three') cy.visit('/board/' + boardId) }) We hope this helps to clarify how console.log() functions work. The id variable, however, is another story. The use of it within the chain appears to be the case. Or is it? Actually, no. Since it is passed as an argument, it is theoretically passed “from outside” and not in the command chain. This variable was set out at the start of the test. In our test, we are instructing Cypress to run the command.visit() with whatever the value of ‘/board/’ + id is. Think about the “inside chain vs. outside chain” theory. Let’s review the code once more: JavaScript it('captures value in variable', () => { // Instant variable declaration, no wait let identifier cy.request('/api/boards') .then( response => { identifier = response.body[0].id }) // Instant variable usage, no wait cy.visit('/board/' + identifier) }) Now that the issue is more evident, let’s examine the many ways we might pass values in our test. Let’s have a look at at least a few of the many solutions that exist for this. Step up the Desired Code in the Command Chain Making ensuring that everything in our command chain is included is the simplest solution. The .visit() function must be used inside the command chain in order to use the updated value. The id will be passed with a new value in this manner. This technique works well when you want to quickly pass a single variable because using multiple .then() functions may result in a “pyramid of doom.” JavaScript it('holds value in variable', () => { let boardId // initialize variable cy.request('/api/boards') .then( response => { boardId = response.body[0].id // set value cy.visit('/board/' + boardId) // use the newly set value }) }) Separate Logic Into Several Tests You can divide the logic into different tests and use a “setup” it() function to assign your variables before executing it() block to use that variable because Cypress executes it() blocks one at a time. This method may be fairly constrained, though, since each variable update requires a separate block. Not every it() function is now a test, so it’s not the ideal test design either. This may also result in a strange domino effect, when the failure of one test may be a result of the failure of another test. Using Hooks Using before() or beforeEach() hooks is a marginally better method of dividing a test. You are more logically dividing your test this way. You have two phases: the preparation process, which is separate from the test, and the execution phase, which is the test itself. This method also has the benefit of giving you explicit information in the error log when a hook fails. Using Aliases and Hooks Aliases are actually a part of the Cypress-bundled Mocha framework, which is used to run tests. Anytime you use the.as() command, an alias will be created in the Mocha context and may be accessed by using this keyword, as demonstrated in the example. It will be a shared variable, allowing you to share variables between tests in the specification. This keyword, however, must be used with the traditional function expression, function(){}, and cannot be used in functions using the arrow expression () =>{}. Take a look at this example. JavaScript beforeEach( function() { cy.request('/api/boards') .as('boardData') }) // using it('utilize variable', () => { ... would not work it('utilize variable', function() { cy.visit('/board/' + this.boardData.body[0].id) }) Using cy.task() A Node.js server process is active in the background behind Cypress. Node can be used, and temporary data can be kept there. If you want, you can even seed a test database! Values from the last run are still present here as long as you don’t close the Cypress console. Why do you do this? You can give a value to Node.js by using the command cy.task(). A “get” and “set” command must be written. This will be a piece of cake if you are comfortable with getters and setters like you are in Java. assignUserId: (value) => { return (userIdentifier = value); } retrieveUserId: () => { return userIdentifier; } cy.get('User').then(($userIdentifier) => { cy.task('assignUserId', $userIdentifier); }); cy.task('retrieveUserId').then((userIdentifier) => { cy.get('User').type(userIdentifier); }); Using Cypress – Fixtures Cypress fixtures are used to maintain and save the test data for automation. The fixtures folder contains the fixtures for the Cypress project (example.json file). Basically, it makes it easier to import data from external files. JavaScript describe('Testing on Browserstack', function () { //part of before hook before(function(){ //retrieve fixture data cy.fixture('sample').then(function(userData){ this.userData = userData }) }) // test case it('Test Scenario 1', function (){ // navigate to URL cy.visit("https://signup.example.com/register/register.php") //data utilized from fixture cy.get(':nth-child(3) > [width="185"] > input') .type(this.userData.userName) cy.get('#mobno').type(this.userData.phoneNumber) }); }); Code reuse is ensured by Cypress, which enables the same test script to be executed against several fixtures files. Sharing Variables Between Test Files Using Environment Variables We can create environment variables that the test automation framework can use globally and that all test cases can access. In our project’s cypress.json file, we can store this kind of customized environment variable. We must specify the key as “env” in the cypress.json file and then set the value because a customized variable is not exposed by default Cypress configurations. In the real test, we must also use Cypress.env and pass the value declared in the json file in order to access this variable. JavaScript describe('ExampleSite Test', function () { // test case it('Scenario A', function (){ // navigate to application using environment variable cy.visit(Cypress.env('siteUrl')) cy.getCookies() cy.setCookie('cookieName', 'cookieValue') }); }); We now know that Cypress is a test automation framework, and much like other test automation frameworks, it must run the same set of tests in a variety of test environments, including DEV, QA, UAT, and others. However, some values or variables, such as the application URL or credentials, could have different values in various test environments. Cypress offers methods for test scripts to access environment variables in order to deal with such circumstances. Environment variables are what Cypress considers to be all the variables within the “env” tag in the config.json file. Below is an example of its syntax: JavaScript { "env": { "api_key": "api_value" } } // Retrieve all the environment variables Cypress.env() // Retrieve a specific environment variable using its key Cypress.env('api_key') Next, let’s consider a different instance. JavaScript { "env": { "Param1": "Data1", "Param2": "Data2" } } Cypress.env(); // {Param1: "Data1", Param2: "Data2"} Cypress.env("Param1"); // It will return "Data1" The “Cypress.env()” method in Cypress can be used to obtain environment variables. Cypress Wrap When utilizing cypress commands like should, type, or click on an object or jquery element, you may first want to wrap it in order to yield the objects that were placed in it and yield its resolved value. JavaScript cy.wrap(entity) cy.wrap(entity, configurations) cy.wrap({Character: 'John'}) const Champion = () => { return 'John' } cy.wrap({ name: Champion }).invoke('name').should('eq', 'John') Hero is a JavaScript object, and Cypress cannot be used to interact with it. We use the key name passed with the object Hero in the wrap to assert that it should be equal to Naruto, and it returns true. We then utilize the wrap to convert the object Hero into Cypress. JavaScript describe('Utilizing Wrap Command', () => { it("Wrapping Various Data Types", () => { // Variable let colorStatement = 'Red Or Blue' cy.wrap(colorStatement).should('eq', 'Red Or Blue') // Object let Character = {name: 'Itachi'} cy.wrap(Character).should('have.property', 'name', 'Itachi') // Array let Characters = ['Itachi', 'Sasuke', 'Naruto', 'Sakura'] cy.wrap(Characters).should('include', 'Sakura') }) })

By Hamid Akhtar

Functional Testing for Ensuring Failsafe Operation of SaaS Applications

SaaS software continues to gain momentum. Experts predict that its global market size will comprise $1.2 trillion in 2032. And it’s not surprising due to the multiple benefits it provides - from smooth scaling to decreased expenses. However, due to high software intricacy, it’s essential to verify its operation thoroughly to make sure no critical defects spoil the user experience or cause malfunction. Therefore, in this article, I’ll highlight the most crucial functional tests that can increase confidence in the software and contribute to achieving business goals. The Essence of SaaS Software From the get-go, SaaS-driven software is designed to be leveraged on the cloud infrastructure. For the target audience, this is very convenient, as it only needs Internet access and any computer, smartphone, or Internet-connected device to effectively utilize the software. Such software must place clients’ security at the core. Although a single instance of the solution is used for each client, sensitive information of all end users is separated and, thus, protected from accidental or deliberate intervention. In addition, these IT products don’t cause any inconvenience for users as all the activities related to ensuring the correct software functioning, providing upgrades, or averting downtimes are taken over by the provider. If businesses leverage SaaS software solutions, they can benefit in many ways, for instance: Cost savings: There’s no need for companies to spend colossal sums on infrastructure, installation, maintenance, and other costly activities. Instant implementation: Right after acquiring the IT product, clients can relish all the opportunities provided by SaaS software rather than potentially wait for weeks for the deployment of on-premises solutions. Increased confidence: Due to automated data recovery opportunities, companies can be sure that all the information is secure. Why Functional Testing Is Beneficial for Saas-Driven Software Despite the multiple advantages provided by SaaS products, it’s important to remember that, like any other software, such applications should be tested properly. Without meticulous QA verifications performed at the earliest SDLC stages, chances are high that defects can slip into the production environment; thus, negatively affecting software operation and even causing downtimes. The cost of such situations is immeasurable: apart from financial losses due to software unavailability, companies can lose end users who are overwhelmed with options nowadays and can simply choose competitors. Moreover, major or critical issues are much longer and costlier to fix post-release because of numerous interdependencies between software modules. This means that extra expenses are inevitable. Such software also often interacts with different external systems and presupposes multiple tailored functionalities for various clients, which only increases the risks of software defects. Functional verification is the most fundamental step in the quality assurance process. It serves as a shield against software defects, helps confirm that IT products’ functionalities operate in strict accordance with specifications, and makes sure that both business and end-user needs are met. Paramount Functional Tests To Drive Desired Business Outcomes To confirm an IT product’s robust operation, QA teams must delve into specifications and business needs, interact with developers and business analysts, define the scope of necessary verifications, write test documentation, and leverage a staging environment to execute functional tests. Although there are 4 levels of sequential testing necessary before software launch — unit, integration, system, and acceptance — in this article I’d like to focus only on 3 of them, as unit tests are most often performed by developers. 1. Integration Testing The operation of SaaS-driven applications can be improved with the addition of multiple functionalities that can be represented by separate modules ― chatbots, voice recording, AR/VR-based modules, reporting, scheduling assistants, and tracking productivity functionality, just to name a few. In addition, such systems are often interconnected with other external solutions, such as CRM, ERP, HRIS, and other software. Their cohesive and joint interaction must be established to avoid glitches or freezes. In this scenario, QA specialists run integration tests to verify interactions and data exchange between different software elements, such as microservices, databases, and external solutions, identify defects related to API interactions, and confirm that business processes that depend on the interaction of diverse software modules work error-free. During testing, QA engineers rely on manifold methods depending on the application architecture, size, software development methodology, and other factors. For instance, they can gauge all IT products’ modules at once, begin with high-level parts, or simultaneously verify modules of a wide variety of levels. 2. System Testing System verifications play a crucial role as they help meticulously verify the operation of the entire product to confirm that all features and integrations work as they should, thus increasing the chances of smooth release and eliminating risks related to post-release expensive rework. During system testing of SaaS software, QA engineers should run several priority tests. First, I’d mention smoke testing. It should be done at the beginning of the testing cycle after a build has been deployed to a test environment. Thus, QA engineers check the main functionality of an IT product and identify critical and major issues that block correct software operation and testing itself. No further testing is performed, if the software doesn’t go through this stage successfully. I also cannot help but mention sanity tests that can follow smoke tests. QA engineers need to fulfill them to confirm that minor code changes haven't negatively affected the correct operation of SaaS software and make sure that separate functions work error-free without focusing on detailed testing of the entire IT solution. One more important aspect relates to end-to-end testing. This is an integral part of verifying the entire system as a whole, including components that end users don’t interact with, during which QA specialists utilize close-to-real user scenarios to validate the behavior of the SaaS software, from start to finish, data workflow throughout the system, the work of UI elements, such as buttons, navigational items, menu bars, notifications, forms for inputting data, and others. Moreover, code updates are indispensable for any software development process, as during the project functionality proliferates and features receive upgrades, patches, and bug fixes. The larger the product, the more often such changes are introduced. This is especially relevant for SaaS-driven software that is known for continuous updates. Although updates are vital for ongoing software enhancement, they can cause problems, especially if novel logic contradicts current software parts. Therefore, QA engineers should perform regression testing to neutralize problems and make sure no new defects have appeared after introduced modifications. During testing, specialists define possible code parts with the largest risk of being influenced and execute tests. 3. Acceptance Testing Creating an IT product that fully meets end-user and business expectations is one of the main goals of any software development process. Even if the software operates error-free but doesn’t fully contribute to solving set tasks, there’s always a possibility that users will choose some other options presented on the market. Therefore, acceptance testing serves to identify issues and gaps in requirements fulfillment and should be done in as close to a real-world environment as possible. For instance, QA engineers make sure that the software is ready for deployment by conducting alpha testing in the development environment based on real-world user scenarios. Intended users or customers may also participate in this process. Afterward, QA engineers can also involve selected groups of real users who take part in beta testing fulfilled in the production environment. P.S. Handy Tips for Bettering QA Effectiveness Usually, functional testing presupposes an extensive scope of verifications that need to be executed, especially considering that SaaS software is growing rapidly and updates and customizations appear on a regular basis. Relying on a manual approach only increases the chance of missing deadlines or compromising overall IT product quality. Therefore, companies can involve QA automation engineers to set up automated testing. With its help, software testing specialists increase the velocity of the QA process by performing more tests in less time, significantly boosting coverage, and increasing accuracy due to neutralizing the risk of human errors. Such scripts can be used repeatedly, which optimizes overall efforts even more. I’d also suggest that companies look closely at shift-left testing. Although expertise is required for its introduction and overcoming some challenges at the beginning, when done properly, it can substantially smooth software releases. This approach helps detect and address issues earlier in the development process when they are less expensive and easier to mitigate, which frees up the time for developers to focus on priority functionality. In addition, if existing QA processes do not bring the desired results, underperformance is observed, or testing costs continuously rise for no apparent reason, I’d recommend reaching out to QA consultants. They can identify reasons for current flaws and suggest steps for turning the tide. On the Way to More Future-Proof and Trouble-Free Software Functional testing of SaaS-driven solutions with intricate logic is a crucial step that helps detect and fix issues early and in a cost-effective manner, ensures stable operation aligned with business and user requirements, and avoids user churn. Depending on the business needs, companies may rely on the most fit-for-purpose testing types, boost confidence in the developed software, and strengthen capabilities in a highly competitive market.

By Pavel Novik

A Comprehensive Guide on Non-Regression Testing

A developer’s code changes may impact the software’s functionality. Even minor changes can have unanticipated consequences or result in the appearance of new bugs. For example, we use regression testing to detect newly discovered problems. Regression testing is the process of re-running tests to ensure that code changes do not affect existing functionality. There are times when there isn’t enough time or resources to run regression tests. A testing team can only examine the system modules modified. They do not perform complete regression testing. This is known as non-regression testing. Non-regression testing is a technique for determining whether a new or modified functionality works properly while assuming that the previous functionality was unaffected. For example, when using non-regression testing, testers examine only the evolving unit or module rather than the entire product, saving resources and time. What Is Non-Regression Testing? Non-regression testing is a technique aimed at verifying whether a new or modified functionality operates correctly with the assumption that the previous functionality wasn’t affected. For example, when applying non-regression testing, testers check only the evolving unit or module instead of the whole product, thus, saving resources and time. Steps of Non-Regression Testing Establish a benchmark software release Define a set of routines capable of stimulating as many software functions as possible Run these routines on both software (the benchmark and the new release under test) and collect data that reflects their behavior. Analyze this data using a post-processing tool to produce statistical results. Report the outcome. Exploratory testing follows similar steps to non-regression testing but differs in its analysis phase and focus, resulting in different results and conclusions. The non-regression testing goal is to see if any undesirable behavior emerges following the most recent software modifications. There, the new application behavior is previously known, allowing the identification of an eventual regression (bug). Exploratory testing, on the other hand, aims to discover how the software works, balancing testing and learning and encouraging testers to create new test cases. Regression and Non-Regression Testing While regression testing aims to ensure that a software bug has been successfully corrected by retesting the modified software, the goal of non-regression testing is to ensure that no new software bugs have been introduced after the software has been updated. In general, this disparity in definitions can be assumed to be based on the results of each test. When a new software version is released with no new features compared to the previous version (i.e., the differences between the two versions are limited to bug fixes or software optimization), both releases are expected to have the same functionalities. In this case, the tests performed on both versions are not likely to produce different results but rather ensure that existing bugs have been fixed and no new bugs have been introduced. This testing methodology distinguishes regression testing. On the other hand, if the new release includes new features or improvements that cause the software to behave differently, the tests performed on the previous and new versions may yield the following results. Desired differences associated with expected new behavior, and undesired differences, indicate a software regression caused by a side-effect bug. Regression Testing vs. Non-Regression Testing A regression test is typically a test that we perform to ensure that the system’s various functionalities are still working as expected and that the new functionalities added did not break any of the existing ones. This could be a combination of API/UI/Unit tests run regularly. Non-regression tests can refer to various things depending on the context of your projects, such as Smoke Testing or Unit Testing that are run during every code check-in. It could also refer to story-level testing carried out when a specific feature/requirement in a story is being tested. Security testing, load testing, and stress testing may also be performed during the development lifecycle. How Non-Regression Testing Is Different From Nonparametric Regression Testing? Nonparametric regression is a type of regression analysis in which the predictor does not take a predetermined form but is built based on data information. That is, no parametric form for the relationship between predictors and the dependent variable is assumed. Nonparametric regression necessitates larger sample sizes than parametric regression because the data must supply both the model structure and the model estimates. Non-Regression Testing Example Non-regression testing can be incorporated into regression analysis. Here’s a rundown of how it works. For example, assume there is an existing functionality A tested and a new functionality B that has just been added to the product. So, for the time being, the program functionality is A+B. Non-regression testing will only cover functionality B. However, a new functionality – C – will be added later. If we want regression testing, we’ll need to run the tests for A and B functionality. Non-regression tests are incorporated into regression testing. Use of Non-Regression Testing Non-regression testing is used when system components evolve, or new system components (and functionality) are added. Its goal is to ensure that the changes are correct and that no regression bugs have appeared in the system due to the recent evolution. In general, previous test sequences are launched to ensure that the system’s testing quality has not deteriorated. The test plan specifies the components to be tested following the evolution or modification of a system component. How To Automate Regression Testing? Regression testing is a multi-layered process due to its extensive coverage and technical complexity. Here’s a step-by-step guide to regression testing and incorporating automation into the workflow. At this stage, a developer estimates which system components will be changed and the extent of the change. Impact analysis of software changes: This stage entails outlining all of the possible consequences of the code change across the system, identifying all systems affected by a new fix or feature, estimating potential system damage, and ways to deal with it. It is creating a strategy for regression testing. The testing team outlines the workflow step by step at this stage. For example, a regression testing strategy might look something like this: 1) collect test data; 2) estimate execution time for test cases; 3) automate test cases; 4) execute tests; 5) report; and 6) iterate. They are developing a test suite. At this point, a QA specialist creates automated tests. Later, the regression when running automated tests on testing automation engineer writes scripts for execution in a scripted language selected by the team ahead of time. Running regression tests When running automated tests, prioritize cases and evaluate test module reusability. Maintain a high frequency of testing and establish a flexible reporting system. Reporting. At this point, QA specialists must explain the testing results to stakeholders such as a project manager, the end client, and anyone who is involved. In addition, a developer must develop the metrics of analysis of the scope of testing and elaborate on how the testing session helped the team achieve a goal set during the planning stage to write a compelling summary report. Conclusion Non-regression testing entails simply testing. Regression testing entails repeatedly testing the application in two scenarios. When a defect is discovered by a tester and corrected by a developer, the tester must focus on the defect functionality and related old functionality. Once the changes have been incorporated into the application, the tester must test both the new and related old functionality to ensure that the related old functionality remains the same. A tester always carries out Non-Regression Testing. If any defects are discovered, then you must perform Regression Testing.

By Yogesh Solanki

Test Parameterization With JUnit 5.7: A Deep Dive Into @EnumSource

Parameterized tests allow developers to efficiently test their code with a range of input values. In the realm of JUnit testing, seasoned users have long grappled with the complexities of implementing these tests. But with the release of JUnit 5.7, a new era of test parameterization enters, offering developers first-class support and enhanced capabilities. Let's delve into the exciting possibilities that JUnit 5.7 brings to the table for parameterized testing! Parameterization Samples From JUnit 5.7 Docs Let's see some examples from the docs: Java @ParameterizedTest @ValueSource(strings = { "racecar", "radar", "able was I ere I saw elba" }) void palindromes(String candidate) { assertTrue(StringUtils.isPalindrome(candidate)); } @ParameterizedTest @CsvSource({ "apple, 1", "banana, 2", "'lemon, lime', 0xF1", "strawberry, 700_000" }) void testWithCsvSource(String fruit, int rank) { assertNotNull(fruit); assertNotEquals(0, rank); } @ParameterizedTest @MethodSource("stringIntAndListProvider") void testWithMultiArgMethodSource(String str, int num, List<String> list) { assertEquals(5, str.length()); assertTrue(num >=1 && num <=2); assertEquals(2, list.size()); } static Stream<Arguments> stringIntAndListProvider() { return Stream.of( arguments("apple", 1, Arrays.asList("a", "b")), arguments("lemon", 2, Arrays.asList("x", "y")) ); } The @ParameterizedTest annotation has to be accompanied by one of several provided source annotations describing where to take the parameters from. The source of the parameters is often referred to as the "data provider." I will not dive into their detailed description here: the JUnit user guide does it better than I could, but allow me to share several observations: The @ValueSource is limited to providing a single parameter value only. In other words, the test method cannot have more than one argument, and the types one can use are restricted as well. Passing multiple arguments is somewhat addressed by @CsvSource, parsing each string into a record that is then passed as arguments field-by-field. This can easily get hard to read with long strings and/or plentiful arguments. The types one can use are also restricted — more on this later. All the sources that declare the actual values in annotations are restricted to values that are compile-time constants (limitation of Java annotations, not JUnit). @MethodSource and @ArgumentsSource provides a stream/collection of (un-typed) n-tuples that are then passed as method arguments. Various actual types are supported to represent the sequence of n-tuples, but none of them guarantee that they will fit the method's argument list. This kind of source requires additional methods or classes, but it provides no restriction on where and how to obtain the test data. As you can see, the source types available range from the simple ones (simple to use, but limited in functionality) to the ultimately flexible ones that require more code to get working. Sidenote — This is generally a sign of good design: a little code is needed for essential functionality, and adding extra complexity is justified when used to enable a more demanding use case. What does not seem to fit this hypothetical simple-to-flexible continuum, is @EnumSource. Take a look at this non-trivial example of four parameter sets with 2 values each. Note — While @EnumSource passes the enum's value as a single test method parameter, conceptually, the test is parameterized by enum's fields, that poses no restriction on the number of parameters. Java enum Direction { UP(0, '^'), RIGHT(90, '>'), DOWN(180, 'v'), LEFT(270, '<'); private final int degrees; private final char ch; Direction(int degrees, char ch) { this.degrees = degrees; this.ch = ch; } } @ParameterizedTest @EnumSource void direction(Direction dir) { assertEquals(0, dir.degrees % 90); assertFalse(Character.isWhitespace(dir.ch)); int orientation = player.getOrientation(); player.turn(dir); assertEquals((orientation + dir.degrees) % 360, player.getOrientation()); } Just think of it: the hardcoded list of values restricts its flexibility severely (no external or generated data), while the amount of additional code needed to declare the enum makes this quite a verbose alternative over, say, @CsvSource. But that is just a first impression. We will see how elegant this can get when leveraging the true power of Java enums. Sidenote: This article does not address the verification of enums that are part of your production code. Those, of course, had to be declared no matter how you choose to verify them. Instead, it focuses on when and how to express your test data in the form of enums. When To Use It There are situations when enums perform better than the alternatives: Multiple Parameters per Test When all you need is a single parameter, you likely do not want to complicate things beyond @ValueSource. But as soon as you need multiple -— say, inputs and expected results — you have to resort to @CsvSource, @MethodSource/@ArgumentsSource or @EnumSource. In a way, enum lets you "smuggle in" any number of data fields. So when you need to add more test method parameters in the future, you simply add more fields in your existing enums, leaving the test method signatures untouched. This becomes priceless when you reuse your data provider in multiple tests. For other sources, one has to employ ArgumentsAccessors or ArgumentsAggregators for the flexibility that enums have out of the box. Type Safety For Java developers, this should be a big one. Parameters read from CSV (files or literals), @MethodSource or @ArgumentsSource, they provide no compile-time guarantee that the parameter count, and their types, are going to match the signature. Obviously, JUnit is going to complain at runtime but forget about any code assistance from your IDE. Same as before, this adds up when you reuse the same parameters for multiple tests. Using a type-safe approach would be a huge win when extending the parameter set in the future. Custom Types This is mostly an advantage over text-based sources, such as the ones reading data from CSV — the values encoded in the text need to be converted to Java types. If you have a custom class to instantiate from the CSV record, you can do it using ArgumentsAggregator. However, your data declaration is still not type-safe — any mismatch between the method signature and declared data will pop up in runtime when "aggregating" arguments. Not to mention that declaring the aggregator class adds more support code needed for your parameterization to work. And we ever favored @CsvSource over @EnumSource to avoid the extra code. Documentable Unlike the other methods, the enum source has Java symbols for both parameter sets (enum instances) and all parameters they contain (enum fields). They provide a straightforward place where to attach documentation in its more natural form — the JavaDoc. It is not that documentation cannot be placed elsewhere, but it will be — by definition — placed further from what it documents and thus be harder to find, and easier to become outdated. But There Is More! Now: Enums. Are. Classes. It feels that many junior developers are yet to realize how powerful Java enums truly are. In other programming languages, they really are just glorified constants. But in Java, they are convenient little implementations of a Flyweight design pattern with (much of the) advantages of full-blown classes. Why is that a good thing? Test Fixture-Related Behavior As with any other class, enums can have methods added to them. This becomes handy if enum test parameters are reused between tests — same data, just tested a little differently. To effectively work with the parameters without significant copy and paste, some helper code needs to be shared between those tests as well. It is not something a helper class and a few static methods would not "solve." Sidenote: Notice that such design suffers from a Feature Envy. Test methods — or worse, helper class methods — would have to pull the data out of the enum objects to perform actions on that data. While this is the (only) way in procedural programming, in the object-oriented world, we can do better. Declaring the "helper" methods right in the enum declaration itself, we would move the code where the data is. Or, to put in OOP lingo, the helper methods would become the "behavior" of the test fixtures implemented as enums. This would not only make the code more idiomatic (calling sensible methods on instances over static methods passing data around), but it would also make it easier to reuse enum parameters across test cases. Inheritance Enums can implement interfaces with (default) methods. When used sensibly, this can be leveraged to share behavior between several data providers — several enums. An example that easily comes to mind is separate enums for positive and negative tests. If they represent a similar kind of test fixture, chances are they have some behavior to share. The Talk Is Cheap Let's illustrate this on a test suite of a hypothetical convertor of source code files, not quite unlike the one performing Python 2 to 3 conversion. To have real confidence in what such a comprehensive tool does, one would end up with an extensive set of input files manifesting various aspects of the language, and matching files to compare the conversion result against. Except for that, it is needed to verify what warnings/errors are served to the user for problematic inputs. This is a natural fit for parameterized tests due to the large number of samples to verify, but it does not quite fit any of the simple JUnit parameter sources, as the data are somewhat complex.See below: Java enum Conversion { CLEAN("imports-correct.2.py", "imports-correct.3.py", Set.of()), WARNINGS("problematic.2.py", "problematic.3.py", Set.of( "Using module 'xyz' that is deprecated" )), SYNTAX_ERROR("syntax-error.py", new RuntimeException("Syntax error on line 17")); // Many, many others ... @Nonnull final String inFile; @CheckForNull final String expectedOutput; @CheckForNull final Exception expectedException; @Nonnull final Set<String> expectedWarnings; Conversion(@Nonnull String inFile, @Nonnull String expectedOutput, @NotNull Set<String> expectedWarnings) { this(inFile, expectedOutput, null, expectedWarnings); } Conversion(@Nonnull String inFile, @Nonnull Exception expectedException) { this(inFile, null, expectedException, Set.of()); } Conversion(@Nonnull String inFile, String expectedOutput, Exception expectedException, @Nonnull Set<String> expectedWarnings) { this.inFile = inFile; this.expectedOutput = expectedOutput; this.expectedException = expectedException; this.expectedWarnings = expectedWarnings; } public File getV2File() { ... } public File getV3File() { ... } } @ParameterizedTest @EnumSource void upgrade(Conversion con) { try { File actual = convert(con.getV2File()); if (con.expectedException != null) { fail("No exception thrown when one was expected", con.expectedException); } assertEquals(con.expectedWarnings, getLoggedWarnings()); new FileAssert(actual).isEqualTo(con.getV3File()); } catch (Exception ex) { assertTypeAndMessageEquals(con.expectedException, ex); } } The usage of enums does not restrict us in how complex the data can be. As you can see, we can define several convenient constructors in the enums, so declaring new parameter sets is nice and clean. This prevents the usage of long argument lists that often end up filled with many "empty" values (nulls, empty strings, or collections) that leave one wondering what argument #7 — you know, one of the nulls — actually represents. Notice how enums enable the use of complex types (Set, RuntimeException) with no restrictions or magical conversions. Passing such data is also completely type-safe. Now, I know what you think. This is awfully wordy. Well, up to a point. Realistically, you are going to have a lot more data samples to verify, so the amount of the boilerplate code will be less significant in comparison. Also, see how related tests can be written leveraging the same enums, and their helper methods: Java @ParameterizedTest @EnumSource // Upgrading files already upgraded always passes, makes no changes, issues no warnings. void upgradeFromV3toV3AlwaysPasses(Conversion con) throws Exception { File actual = convert(con.getV3File()); assertEquals(Set.of(), getLoggedWarnings()); new FileAssert(actual).isEqualTo(con.getV3File()); } @ParameterizedTest @EnumSource // Downgrading files created by upgrade procedure is expected to always pass without warnings. void downgrade(Conversion con) throws Exception { File actual = convert(con.getV3File()); assertEquals(Set.of(), getLoggedWarnings()); new FileAssert(actual).isEqualTo(con.getV2File()); } Some More Talk After All Conceptually, @EnumSourceencourages you to create a complex, machine-readable description of individual test scenarios, blurring the line between data providers and test fixtures. One other great thing about having each data set expressed as a Java symbol (enum element) is that they can be used individually; completely out of data providers/parameterized tests. Since they have a reasonable name and they are self-contained (in terms of data and behavior), they contribute to nice and readable tests. Java @Test void warnWhenNoEventsReported() throws Exception { FixtureXmls.Invalid events = FixtureXmls.Invalid.NO_EVENTS_REPORTED; // read() is a helper method that is shared by all FixtureXmls try (InputStream is = events.read()) { EventList el = consume(is); assertEquals(Set.of(...), el.getWarnings()); } } Now, @EnumSource is not going to be one of your most frequently used argument sources, and that is a good thing, as overusing it would do no good. But in the right circumstances, it comes in handy to know how to use all they have to offer.

By Oliver Gondza

Black Box Tester in Python

Brief Problem Description Imagine the situation: you (a Python developer) start a new job or join a new project, and you are told that the documentation is not up to date or is even absent, and those, who wrote the code, resigned a long time ago. Moreover, the code is written in a language that you are not familiar with (or “that you do not know”). You open the code, start examining it, and realize that there are no tests either. Also, the service has been working on Prod for so long that you are afraid to change something. I am not talking about any particular project or company. I have experienced this at least three times. Black Box Well, you have a black box that has API methods (judging by the code) and you know that it pulls something and writes to a database. There is also documentation for those services that are receiving requests. The advantages include the fact that it starts there is documentation on the API that it pulls, and the service code is quite readable. As for the disadvantages, it wants to get something via API. Something can be run in a container, and something can be used from a developer environment, but not everything. Another problem is that requests to the black box are encrypted and signed as well as requests from it to some other services. At the same time, you need to change something in this service and not break what is working. In such cases, Postman or cURL is inconvenient to use. You need to prepare each request in each specific case since there are dynamic input data and signatures that depend on the time of the request.There are almost no ready-made tests, and it is difficult to write them if you do not know the language very well. The market offers solutions that allow you to run tests in such a service. However, I have never used them, so trying to understand them would be more difficult and would take much more time than creating my own solution. Created Solution I have come up with a simple and convenient option. I have written a simple script in Python that will pull this very application. I used requests and a simple signature that I created very quickly for the requests prepared in advance.Next, I needed to mock backends. First Option To do this, I just ran a mock service in Python. In my case, Django turned out to be the fastest and easiest tool for this. I decided to implement everything as simply and quickly as possible and used the latest version of Django. The result was quite good, but it was only one method and it took me several hours to use despite the fact that I wanted to save time. There are dozens of such methods. Examples of Configuration Files In the end, I got rid of everything I did not need and simply generated JSON with requests and responses. I described each request from the front end of my application, the expected response of the service to which requests were sent, as well as the rules for checking the response to the main request.For each method, I wrote a separate URL. However, manually changing the responses of one method from correct to incorrect and vice versa and then pulling each method is difficult and time-consuming. JSON { "id": 308, "front": { "method": "/method1", "request": { "method": "POST", "data": { "from_date": "dfsdsf", "some_type": "dfsdsf", "goods": [ { "price": "112323", "name": "123123", "quantity": 1 } ], "total_amount": "2113213" } }, "response": { "code": 200, "body": { "status": "OK", "data": { "uniq_id": "sdfsdfsdf", "data": [ { "number": "12223", "order_id": "12223", "status": "active", "code": "12223", "url": "12223", "op_id": "12223" } ] } } } }, "backend": { "response": { "code": 200, "method": "POST", "data": { "body": { "status": 1, "data": { "uniq_id": "sdfsdfsdf", "data": [ { "number": "12223", "order_id": "12223", "status": "active", "code": "12223", "url": "12223", "op_id": "12223" } ] } } } } } } Second Option Then I linked mock objects to the script. As a result, it appeared that there is a script call that pulls my application and there is a mock object that responds to all its requests. The script saves the ID of the selected request, and the mock object generates a response based on this ID. Thus, I collected all requests in different options: correct and with errors. What I Got As a result, I got a simple view with one function for all URLs. This function takes a certain request identifier and, based on it, looks for the response rules — a mock object. In the meantime, the script that pulls the service before the request writes this very request identifier to the storage. This script simply takes each case in turn, writes an identifier, and makes the correct request, then it checks if the response is correct, and that's it. Intermediate Connections However, I needed not only to generate responses to these requests but also to test requests to mock objects. After all, the service could send an incorrect request, so it was necessary to check them too. As a result, there was a huge number of configuration files, and my several API methods turned into hundreds of large configuration files for checking. Connecting Database I decided to transfer everything to a database. My service began to write not only to the console but also to the database so that it would be possible to generate reports. That appeared to be more convenient: each case had its own entry in the database. Cases are combined into projects and have flags that allow you to disable irrelevant options. In the settings, I added request and response modifiers, which should be applied to each request and response at all levels. To simplify this as much as possible, I use SQLite. Django has it by default. I have transferred all configuration files to the database and saved all testing results in it. Algorithm Therefore, I found a very simple and flexible solution. It already works as an external integration test for three microservices, but I am the only one who uses it. It certainly does not override unit tests, but it complements them well. When I need to validate services, I use this Django tester to do that. Configuration File Example The settings have become simpler and are managed with Django Admin. I can easily turn them off, change, and watch history. I could go further and make a full-fledged UI, but this is more than enough for me for now. Request Body JSON JSON { "from_date": "dfsdsf", "some_type": "dfsdsf", "goods": [ { "price": "112323", "name": "123123", "quantity": 1 } ], "total_amount": "2113213" } Response Body JSON JSON { "uniq_id": "sdfsdfsdf", "data": [ { "number": "12223", "order_id": "12223", "status": "active", "code": "12223", "url": "12223", "op_id": "12223" } ] } Backend Response Body JSON JSON { "status": 1, "data": { "uniq_id": "sdfsdfsdf", "data": [ { "number": "12223", "order_id": "12223", "status": "active", "code": "12223", "url": "12223", "op_id": "12223" } ] } } What It Gives You In what way can this service be useful? Sometimes, even with tests, you need to pull services from the outside, or several services in a chain. Services can also be black boxes. A database can be run in Docker. As for an API...an API can be run in Docker as well. You need to set a host, port, and configuration files and run it. Why the Unusual Solution? Some may say that you can use third-party tools integration tests or some other tests. Of course, you can! But, with limited resources, there is often no time to apply all this, and quick and effective solutions are needed. And here comes the simplest Django service that meets all requirements.

By Aleksei Sharypov

How To Run OWASP ZAP Security Tests in Azure DevOps Pipeline

Security testing is an essential part of testing. Every organization wants to do at least basic security testing before releasing the code to production. Security testing is like an ocean; it might be difficult to perform complete security testing without the help of trained professionals. Some of the open-source tools provide automated basic scanning of the website. Once we add it to pipelines like any other test such as smoke or regression, the security tests also can run as part of deployment and report issues. What Is OWASP ZAP? ZAP is a popular security testing tool and open source. ZAP tool helps to find the vulnerabilities in the applications or API endpoints. Vulnerabilities include cross-site scripting, SQL injection, broken authentication, sensitive data exposure, broken access control, security misconfiguration, insecure deserialization, etc. The beauty of this tool is that it provides both UI and Command Line Interfaces to run the tests. Since it provides a command-line interface we can integrate it as part of our pipeline. The pipeline can be triggered when we release code into production, this helps to find the potential security issues. What Are We Going To Learn? How to configure and set up OWASP ZAP security test into Azure Release Pipeline How to run OWASP ZAP security tests on websites in Azure DevOps Pipeline using Docker How to perform API security testing using OWASP ZAP security testing tool in Azure DevOps Pipelines with Docker Images How to publish OWASP ZAP security testing results in Azure DevOps Pipeline How to publish OWASP ZAP HTML test results into Azure Artifacts by creating feed and packages How to download artifacts containing OWASP ZAP HTML test results using the Azure CLI tool What Are the Prerequisites? Create a Repository Create a repository inside your organization (preferred), download the file OWASPToNUnit3.xslt, and keep it inside the repository. This file is needed to convert the OWASP ZAP security test result XML file to publish results in Azure DevOps. Create a Feed Azure DevOps Artifact This feed is helpful for publishing OWASP ZAP HTML results. The steps are as follows: Step 1 Navigate to Azure DevOps > Click on Artifacts > Click on Create Feed: Step 2 In the "Create new feed" form, enter the correct text, and click on Create.Note: We will be using the feed name while configuring tasks. You need to choose the same from the drop-down, so note down the feed name. Step 3 Create a sample package inside the feed using the command line. Install Azure CLI. After installation, run the command below to create a sample package: PowerShell az artifacts universal publish - -organization https://dev.azure.com/[Your_Org_Name] --feed SecurityTesting --name security_testing --version 1.0.0 --description "Your description" --path . Upon completion of Step 3, Navigate to Azure DevOps > Artifact > and select feed as SecurityTesting. You should see the newly created package: We have completed all initial setup and prerequisites, and are good to start with pipelines now. Refer to Microsoft Documentation for more details. How to Configure OWASP ZAP Security Tests in Azure DevOps Pipeline Let's discuss in detail step by step by setting up OWASP ZAP Security Tests Pipeline using Docker Image. Step 1: Create a New Release Pipeline 1. Navigate to Azure DevOps > Pipeline > click on Releases. 2. Click on New, and choose New Release Pipeline: 3. Choose Empty job when the template window prompts: 4. Name the stage Security Testing (or any other name you wish). Step 2: Add Artifact to Release Pipeline Click on Add an artifact. In the popup window, choose Azure Repository. Choose your Project. Choose the Source repository (this is the place where you created the XSLT file in the prerequisite section). Choose the default branch as master. Click Add. Step 3: Add Tasks to Pipeline We need to add tasks to the pipeline. In our case, we have created only one stage, which is security testing. Step 4: Configure Agent Job Details Display Name: Agent Job or anything you wish Agent pool: Choose Azure Pipelines. Agent Specification: Choose any Ubuntu agent from the dropdown. Step 5: Add Docker Installer Task In the search box, search for Docker CLI, Add the task, and configure the Docker CLI Task. Step 6: Add Bash Script Task Step 7: Configure Bash Script Task Enter display name: Security Test Run Type: Click on the Inline Radio button. Script: Copy and paste the below code (don't forget to replace your URL). Example: chmod -R 777 ./ docker run --rm \ -v $(pwd):/zap/wrk/:rw \ -t owasp/zap2docker-stable \ zap-full-scan.py \ -t https://dzone.com \ -g gen.conf \ -x OWASP-ZAP-Report.xml \ -r scan-report.html How To Run OWASP ZAP Security Test for API The above-mentioned script works well with websites and webpages, but if your requirement is an API, then you need to add different inline scripts. The rest of the things remain the same. Script for OWASP ZAP API Security Scan Shell chmod -R 777 ./ docker run — rm -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-weekly zap-api-scan.py -t [your-api-url] -f openapi -g api-scan.conf -x OWASP-ZAP-Report.xml -r api-scan-report.html true Example: chmod -R 777 ./ docker run --rm \ -v $(pwd):/zap/wrk/:rw \ -t owasp/zap2docker-weekly \ zap-api-scan.py \ -t https://dzone.com/swagger/v1/swagger.json \ -f openapi \ -g api-scan.conf \ -x OWASP-ZAP-Report.xml \ -r api-scan-report.html true Thanks to sudhinsureshr for this. Step 8: Add Powershell Task To Convert ZAP XML Report To Azure DevOps NUnit Report Format To Publish Results Add PowerShell task using the add Azure DevOps/add tasks window. Configure Powershell task. Convert ZAP XML to NUnit XML. Display Name: Anything you wish Type: Inline Script: Inline Sample Inline Script Note: This script contains a relative path to the repository and folder. The content of the script may change based on the name you specified in your project. PowerShell $XslPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)/_Quality/SecurityTesting/OWASPToNUnit3.xslt" $XmlInputPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)/OWASP-ZAP-Report.xml" $XmlOutputPath = "$($Env:SYSTEM_DEFAULTWORKINGDIRECTORY)/Converted-OWASP-ZAP-Report.xml" $XslTransform = New-Object System.Xml.Xsl.XslCompiledTransform $XslTransform.Load($XslPath) $XslTransform.Transform($XmlInputPath, $XmlOutputPath) Step 9: [Optional] Publish OWASP ZAP Security Testing HTML Results To Azure Artifact Add Universal Package task: Configure Universal Package task: Display Name: Anything you wish Command: Publish (Choose from the dropdown) Path to Publish: $(System.DefaultWorkingDirectory) or you can choose from the selection panel (…) menu Feed Location: This organization's collection Destination Feed: SecurityTesting (This is the one that you created in prerequisite step 2.) Package Name: security_testing (This is the one that you created in prerequisite step 3.) Step 10: Publish OWASP ZAP Results Into Azure DevOps Pipeline Add Publish Results task: Configure Publish Results Task Display Name: Any name Test Result format: NUnit Test Result Files: Output file name in Step 8. In our case, it's Converted-OWASP-ZAP-Report.xml. Search Folder: $(System.DefaultWorkingDirectory) After completion of Step 10, trigger Azure OWASP ZAP release. The release starts running and shows the progress in the command line. Step 11: Viewing OWASP/ZAP Security Testing Results Once the release is completed, navigate to completed tasks and click on the Publish Test Results task. The window with the link to the result opens: Once you click the link, you can see the results. Final Thoughts ZAP is an acronym for Zed Attack Proxy, formerly known as OWASP ZAP. It is primarily used as a web application security scanner. The goal is to find vulnerabilities in an application or API endpoint that are prone to various types of attacks. ZAP is actively maintained by a dedicated team of volunteers and is used extensively by professional penetration testers. As we can see in this article, the detailed configuration steps to set up security testing can be added to the DevOps pipeline just like any other tests, and run as a part of deployment and report issues.

By Ganesh Hegde

CORE

Long Tests: Saving All App’s Debug Logs and Writing Your Own Logs

Let's imagine we have an app installed on a Linux server in the cloud. This app uses a list of user proxies to establish an internet connection through them and perform operations with online resources. The Problem Sometimes, the app has connection errors. These errors are common, but it's unclear whether they stem from a bug in the app, issues with the proxies, network/OS conditions on the server (where the app is running), or just specific cases that don't generate a particular error message. These errors only occur sometimes and not with every proxy but with many different ones (SSH, SOCKS, HTTP(s), with and without UDP), providing no direct clues that the proxies are the cause. Additionally, it happens at a specific time of day (but this might be a coincidence). The only information available is a brief report from a user, lacking details. Short tests across different environments with various proxies and network conditions haven’t reproduced the problem, but the user claims it still occurs. The Solution Rent the same server with the same configuration. Install the same version of the app. Run tests for 24+ hours to emulate the user's actions. Gather as much information as possible (all logs – app logs, user (test) action logs, used proxies, etc.) in a way that makes it possible to match IDs and obtain technical details in case of errors. The Task Write some tests with logs. Find a way to save all the log data. To make it more challenging, I'll introduce a couple of additional obstacles and assume limited resources and a deadline. By the way, this scenario is based on a real-world experience of mine, with slight twists and some details omitted (which are not important for the point). Testing Scripts and Your Logs I'll start with the simplest, most intuitive method for beginner programmers: when you perform actions in your scripts, you need to log specific information: Python output_file_path = "output_test_script.txt" def start(): # your function logic print(f'start: {response.content}') with open(output_file_path, "a") as file: file.write(f'uuid is {uuid} -- {response.content} \n') def stop(): # your function logic print(local_api_data_stop, local_api_stop_response.content) with open(output_file_path, "a") as file: file.write(f'{uuid} -- {response.content} \n') # your other functions and logic if __name__ == "__main__": with open(output_file_path, "w") as file: pass Continuing, you can use print statements and save information on actions, responses, IDs, counts, etc. This approach is straightforward, simple, and direct, and it will work in many cases. However, logging everything in this manner is not considered best practice. Instead, you can utilize the built-in logging module for a more structured and efficient logging approach. Python import logging # logger object logger = logging.getLogger('example') logger.setLevel(logging.DEBUG) # file handler fh = logging.FileHandler('example.log') fh.setLevel(logging.DEBUG) # formatter, set it for the handlers formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') fh.setFormatter(formatter) ch.setFormatter(formatter) # add the handlers to the logger logger.addHandler(fh) logger.addHandler(ch) # logging messages logger.debug('a debug message') logger.info('an info message') logger.warning('a warning message') logger.error('an error message') Details. The first task is DONE! Let's consider a case where your application has a debug log feature — it rotates with 3 files, each capped at 1MB. Typically, this config is sufficient. But during extended testing sessions lasting 24hrs+, with heavy activity, you may find yourself losing valuable logs due to this configuration. To deal with this issue, you might modify the application to have larger debug log files. However, this would necessitate a new build and various other adjustments. This solution may indeed be optimal, yet there are instances where such a straightforward option isn’t available. For example, you might use a server setup with restricted access, etc. In such cases, you may need to find alternative approaches or workarounds. Using Python, you can write a script to transfer information from the debug log to a log file without size restrictions or rotation limitations. A basic implementation could be as follows: Python import time def read_new_lines(log_file, last_position): with open(log_file, 'r') as file: file.seek(last_position) new_data = file.read() new_position = file.tell() return new_data, new_position def copy_new_logs(log_file, output_log): last_position = 0 while True: new_data, last_position = read_new_lines(log_file, last_position) if new_data: with open(output_log, 'a') as output_file: output_file.write(new_data) time.sleep(1) source_log_file = 'debug.log' output_log = 'combined_log.txt' copy_new_logs(source_log_file, output_log) Now let's assume that Python isn't an option on the server for some reason — perhaps installation isn't possible due to time constraints, permission limitations, or conflicts with the operating system and you don’t know how to fix it. In such cases, using bash is the right choice: Python #!/bin/bash source_log_file="debug.log" output_log="combined_log.txt" copy_new_logs() { while true; do tail -F -n +1 "$source_log_file" >> "$output_log" sleep 1 done } trap "echo 'Interrupted! Exiting...' && exit" SIGINT copy_new_logs The second task is DONE! With your detailed logs combined with the app's logs, you now have comprehensive debug information to understand the sequence of events. This includes IDs, proxies, test data, etc along with the actions taken and the used proxies. You can run your scripts for long hours without constant supervision. The only task remaining is to analyze the debug logs to get statistics and potential info on the root cause of any issues, if they even can be replicated according to user reports. Some issues required thorough testing and detailed logging. By replicating the users’ setup and running extensive tests, we can gather important data for pinpointing bugs. Whether using Python or bash scripts (or any other PL), our focus on capturing detailed logs enables us to identify the root causes of errors and troubleshoot effectively. This highlights the importance of detailed logging in reproducing complex technical bugs and issues.

By Konstantin Sakhchinskiy