valFuzz Documentation

            _ _____
__   ____ _| |  ___|   _ ________
\ \ / / _` | | |_ | | | |_  /_  /
 \ V / (_| | |  _|| |_| |/ / / /
  \_/ \__,_|_|_|   \__,_/___/___|
 
A modern testing & fuzzing library for C++
Settings:
 - Multithreaded: 1
 - Max threads: 4
 - Run Fuzzs: 0
 - Run Benchmarks: 0
 - Verbose: 0
 
Before test
Seed: 1730108360
Running 1337 tests...

Welcome to the official valFuzz documentation. This guide provides comprehensive information and practical examples to help you use valFuzz for your testing needs.

Overview

valFuzz is a lightweight testing, fuzzing and benchmarking framework for C++17 onwards. It uses modern standards and techniques to provide a robust and efficient framework for testing and fuzzing for small-medium size projects. The simpliciry of the library allows for easy extension and customization for your own need.

Installation

You can compile the library as a shared library or a static library using cmake. valfuzz has no dependencies other than the standard library.

CMake

cmake -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4

Usage

Assertions

The library provides various macros for assertions to be used in your tests:

• ASSERT - Asserts that the given expression is true.
• ASSERT_EQ - Asserts that the two given values are equal.
• ASSERT_NE - Asserts that the two given values are not equal.
• ASSERT_LT - Asserts that the first value is less than the second value.
• ASSERT_LE - Asserts that the first value is less than or equal to the second value.
• ASSERT_GT - Asserts that the first value is greater than the second value.
• ASSERT_GE - Asserts that the first value is greater than or equal to the second value.
• ASSERT_THROW - Asserts that the given expression throws an exception of the given type.
• ASSERT_NO_THROW - Asserts that the given expression does not throw any exception.

For example:

ASSERT(1 == 1);  // Evaluates the expression
ASSERT_NE(1, 2); // Evaluates the two expressions and compares them
ASSERT_THROW(foo(), std::runtime_error); // Evaluates the expression and checks if it throws the given exception

When an assertion fails, the program will output the test name, the line number and the assertion arguments that caused the failure. The program will then continue to run the other tests.

test: Simple Assertion Fail, line: 66, Assertion failed: 1 == 2
test: Should Not Throw Fail, line: 85, Exception thrown

Your First Test

To run your first test, you need to define a test function using the TEST macro. You will have to specify an unique name for the test and a string name for display purposes.

#include "valfuzz/valfuzz.hpp"
 
TEST(simple_assertion, "Simple Assertion")
{
   ASSERT(1 == 1);
}

Tests are automatically registered and run when the program starts. You can run a particular test by specifying the test name after --test as a command line argument:

./my_test --test "Simple Assertion"

Setup and Cleanup

It is often necessary to perform some setup and cleanup operations before tests are run. You can define those functions using the BEFORE and AFTER macros respectively. The setup function will be run before all tests and the cleanup function will be run after all tests.

#include "valfuzz/valfuzz.hpp"
 
BEFORE()
{
   std::cout << "Before test" << std::endl;
}
 
TEST(a_test, "Another simple test")
{
   ASSERT_EQ(1, 1);
}
 
AFTER()
{
   std::cout << "After test" << std::endl;
}

Fuzzing

valFuzz provides a simple interface for fuzzing your code. You can define a fuzz function using the FUZZME macro, this function will be called repedetly on multiple threads until you stop the program. You can disable multithreading by passing --no-multithread as a command line argument.

FUZZME(naive_fuzzing, "Naive fuzzing")
{
    ASSERT_EQ(1==1);
}

You can get random values using the T get_random<T>() function, where T is the type of the value you want to get. The function will return a random value of the given type, allowed types are:

• int
• float
• double
• char
• std::string

FUZZME(divide_fuzzing, "Divide fuzzing")
{
    int a = valfuzz::get_random<int>();
    int b = valfuzz::get_random<int>();
    int ret = foo_bar(a, b);
    ASSERT_NO_THROW(ret);
}

After defining the fuzz function, you can run it by passing --fuzz as a command line argument:

./my_test --fuzz

Benchmarking

You can define a benchmark function with the macro BENCHMARK. Inside the benchmark, you can do any setup / cleanup and run an expression to be benchmarked with BENCHMARK_RUN.

The first argument to BENCHMARK_RUN is the input size, and the second argument is the expression to be benchmarked. The input size is specified for logging purposes only and can be set to any value that can be printed with std::cout.

‍currently, the cache will be reset between benchmarks only on linux

For example:

int sum_slow(int a, int b)
{
    int sum = 0;
    for (int i = 0; i < a; i++)
    {
        sum += b;
    }
    return sum;
}
 
BENCHMARK(bench_sum_slow, "Sum slow")
{
    int a = 1000;
    int b = 2000;
    RUN_BENCHMARK(a, sum_slow(a, b));
}

Compile and run with --benchmark flag:

./build/valfuzz --benchmark --verbose

Running benchmark: Sum slow
benchmark: "Sum slow"
 - space: 10000
 - min: 1.7115e-05s
 - max: 9.2012e-05s
 - mean: 1.93739e-05s
 - standard deviation: 3.02685e-06
 - Q1: 1.8363e-05s
 - Q3: 2.0115e-05s

The benchmark will be run 100000 times to get the average time. You can set the number of iterations using the --num-iterations flag.

Save a Benchmark Report

You can save the results of your benchmark in a CSV file by passing --report <file> as a command line argument. The output will be saved using the following format:

name,space,min,max,mean,sd,q1,q3

Plotting

After you have collected the benchmark data, you can generate plots and graphs using python templates and examples provided in the plotting directory. The templates use seaborn and matplotlib libraries to generate the graphs.

Multithreading

All tests and fuzz functions are run on multiple threads by default. You can disable multithreading by passing --no-multithread as a command line argument.

Command Line Arguments

You can pass the following command line arguments to the program:

• --test <test_name> - Run a specific test by name.
• --fuzz - Run all fuzz functions.
• --fuzz <fuzz_name> - Run a specific fuzz function by name.
• --benchmark - Run benchmarks.
• --num-iterations <num> - Set the number of iterations for benchmarks.
• --run-one-benchmark <name> - run a specific benchmark
• --no-multithread - Disable multithreading.
• --verbose - Enable verbose output.
• --max-threads <n> - Set the maximum number of threads to use.
• --no-header - Disable the header print at the start.
• --seed <seed> - Set the seed for the random number generator.
• --report <file> - Save benchmark report to a file.
• --help - Display help information.