Go Tracing Function Arguments in Production

Kubernetes is an incredibly complex system for managing and orchestrating containerized applications. It provides a way to run and manage your applications across multiple machines, abstracting away the underlying infrastructure complexities. With Kubernetes, you can easily deploy, scale, and update your applications, ensuring high availability, fault tolerance, and efficient resource utilization.

With so many moving parts, it can be difficult to pinpoint performance issues and troubleshoot problems when they arise. This is where eBPF (extended Berkeley Packet Filter) earns its place. It can trace system events in real time and provide detailed insight into system behaviour, without you having to add log lines or restart anything. In this post we will explore how eBPF can be used to trace Go function arguments in a Kubernetes environment, and why that visibility is so valuable in production.

What is eBPF?

eBPF (extended Berkeley Packet Filter) is a technology that lets you run small, sandboxed programs inside the Linux kernel without changing kernel source or loading kernel modules. It began as a way to filter network packets and has grown into a general mechanism for deep observability, including observing Go processes with eBPF, making it easier to trace behavior at both the system level and the application level. Because the program runs in the kernel, it sees what is happening with very low overhead, which is exactly what you want on a live system you cannot afford to slow down.

Why Trace Go Function Arguments?

In a Kubernetes environment, applications are often written in Go, a popular language for building containerized services. When performance issues arise, it can be challenging to pinpoint the root cause, because logs only show what the developer thought to record. By tracing the arguments passed into a Go function, we capture the actual values flowing through the application at runtime. That turns debugging from guesswork into evidence: instead of wondering what triggered the slow path, you can see the inputs that led to it.

Tracing Go Function Arguments with eBPF

eBPF lets us insert probes into kernel or user-space code and capture data about the system’s behavior as it runs. Before we start, make sure you have the bcc toolkit installed, since it gives us a friendly way to write and load eBPF programs. Installation instructions are in the bcc documentation.

The Example Go Program

Let’s start with a deliberately tiny program so the tracing is easy to follow. It just adds the two numbers passed as arguments.

//main.go
package main

import (
    "fmt"
)

func add(a, b int) int {
    return a + b
}

func main() {
    result := add(2, 3)
    fmt.Println("Result:", result)
}

Writing the eBPF Program

Now the eBPF program that captures the arguments to the function we trace. It reads the first two parameters straight from the CPU registers and prints them.

//trace_args.c
#include <uapi/linux/ptrace.h>

BPF_PERF_OUTPUT(events);

int trace_func_args(struct pt_regs *ctx) {
    int arg1 = PT_REGS_PARM1(ctx);
    int arg2 = PT_REGS_PARM2(ctx);

    bpf_trace_printk("arg1: %d, arg2: %d\n", arg1, arg2);

    return 0;
}

The PT_REGS_PARM1 and PT_REGS_PARM2 macros pull the first and second arguments out of the register context that the kernel hands to the probe. Save the file as trace_args.c.

Compiling the eBPF Program

Compile the eBPF program using clang, targeting the bpf architecture:

clang -O2 -target bpf -c trace_args.c -o trace_args.o

Loading and Attaching From Go

Once we have written our eBPF program, we modify the Go code to load the compiled object and attach it so that our probe fires when the target function runs.

//main.go
package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "unsafe"

    "golang.org/x/sys/unix"
)

const (
    perfEventTypeTracepoint = 10
    tracepointCategory      = "go"

    bpfProgramPath = "./trace_args.o"
)

func loadBPFProgram() int {
    fd, err := unix.BPF(unix.BPF_PROG_LOAD, bpfProgramPath)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to load BPF program: %v\n", err)
        return -1
    }
    return fd
}

func attachTracepoint(fd int, tracepoint string) int {
    attr := &unix.PerfEventAttr{
        Type:   perfEventTypeTracepoint,
        Config: uint64(fd),
    }
    attr.SetSampleFreq(1)
    attr.SetWakeupEvents(1)

    tpName := fmt.Sprintf("%s:%s", tracepointCategory, tracepoint)
    cTpName := unix.ByteSliceFromString(tpName)
    attr.SetTracepoint(unix.BytePtrFromString(tpName))

    _, _, errno := unix.Syscall6(
        unix.SYS_PERF_EVENT_OPEN,
        uintptr(unsafe.Pointer(attr)),
        unix.Getpid(),
        -1,
        -1,
        unix.PERF_FLAG_FD_CLOEXEC,
    )
    if errno != 0 {
        fmt.Fprintf(os.Stderr, "Failed to attach to tracepoint %s: %v\n", tracepoint, errno)
        return -1
    }
    return 0
}

func main() {
    fd := loadBPFProgram()
    if fd < 0 {
        os.Exit(1)
    }
    defer unix.Close(fd)

    if attachTracepoint(fd, "trace_func_args") < 0 {
        os.Exit(1)
    }

    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
    <-signalChan

    fmt.Println("Exiting...")
}

With our eBPF program running, we can send requests to our application and capture the function arguments in real time. The loader program stays alive until you interrupt it, keeping the probe attached while your application handles traffic.

Running It and Reading the Output

Now let’s compile and run the modified Go code:

go build -o main main.go
sudo ./main

When you run the code, it traces the add function arguments using the eBPF program and prints them. Press Ctrl+C to stop the program. The output will be similar to:

arg1: 2, arg2: 3

Those are the exact values passed into add at runtime, read straight from the kernel rather than from anything the application chose to log.

Why This Matters in Production

The reason this technique is powerful is that it is non-intrusive. You are not adding log lines, rebuilding the binary, or restarting the service. The application keeps running exactly as it is, while eBPF observes it from the kernel. In production, where you often cannot reproduce a problem locally and cannot afford downtime to instrument it, that combination of live visibility and near-zero overhead is what makes eBPF so compelling. The same low-overhead approach to capturing real behavior underpins how Keploy handles API testing, generating tests and mocks from real traffic instead of hand-written fixtures.

Conclusion

eBPF is a powerful tool for tracing system events in real time and gaining detailed insight into system behaviour. By tracing Go function arguments in a Kubernetes environment, we get a direct window into what an application is actually doing, argument values and all, without touching the application code. That visibility can be the difference between guessing at a performance issue and seeing its cause, which in turn helps you keep your applications running at their best.

Frequently Asked Questions

What is eBPF used for?

eBPF runs small, sandboxed programs inside the Linux kernel to observe and act on system and application behavior in real time. It is widely used for networking, security, and observability, including tracing function calls and their arguments with very low overhead.

Why trace function arguments instead of just reading logs?

Logs only show what the developer chose to record. Tracing arguments with eBPF captures the actual values passed into a function at runtime, which is invaluable when a performance issue depends on specific inputs that were never logged.

Does eBPF tracing slow down my application?

eBPF is designed for minimal overhead and runs inside the kernel without modifying your application. This makes it suitable for production use, where you need visibility without restarting services or degrading performance.

What tools do I need to start tracing Go functions with eBPF?

You need the bcc toolkit to write and load eBPF programs, clang to compile them into bpf objects, and a Linux environment with eBPF support. From there you can attach probes to the Go functions you want to observe.