⚡ Cribl onboarding 10 times faster: building pipelines with AI

Anyone who implements Cribl for a client knows the platform itself is often up and running quickly. Most of the time goes into onboarding data sources. Every source has its own log format and needs a pipeline that parses the data, cleans it up, enriches it and prepares it for the destination. From firewalls and proxies to Windows, Linux, cloud services and applications. Before you know it, you have dozens of pipelines to build.

I used to build those pipelines by hand, sometimes starting from existing Cribl packs. Collecting and analysing log samples, configuring functions, writing regex, testing and adjusting. A time-consuming process that delivered only one or two working pipelines a week on average.

These days I use Claude Code, and within five minutes I can have it generate a pipeline. Built on synthetic test data, it then works for 70–80% once we connect the real data later; built on good, representative client data, the generated pipeline often works fully right away. Because it works from fixed standards, rules and memory, the pipelines are also built more consistently, of better quality and well documented.

In this post I show how I use Claude Code for this, how I set it up and what to watch out for.

Below I walk through the workflow step by step. First I set the rules and make sure I have usable data without client details. Then I build the first pipeline and test it in Cribl. Every pipeline after that builds on the project's memory and goes into Cribl as well. Step 6 sits outside that sequence: a separate agent that looks across the whole process.

Step 1: a project with clear rules

I work in Claude Code, with a project folder per client. It holds all pipelines, samples and lookups, plus one instructions file: CLAUDE.md. Claude reads that file at the start of every session, and it is where I spend most of my time. It covers things like:

• What to do and what not to do. Which Cribl functions I use and which I avoid.
• My style. How I organise functions and groups. Every pipeline gets a comment with the latest change and who worked on it. Every function gets its own short comment explaining why, not just what.
• How to answer. Briefly, with the files as the result, and ask first when something is unclear.
• How to handle assumptions and/or guessing. Don't. If the AI isn't sure about a field name or format, it says so and asks. An assumption must be recognisable as an assumption.
• Which sources to use, and which not. For information: the Cribl documentation and the official documentation of the log source itself. Not random forums or blogs as a source of truth.

Tip: don't only write down how the AI should do things. It is often even more important to write down what it should not do.

Step 2: data, without real client details

Then the most important rule of thumb: use AI within the rules of the organisation you work for, and never send real client log samples to a publicly accessible AI service. No hostnames, no IP addresses, no usernames, no email addresses and no other personal data. If a company runs its own closed AI environment, the rules there may be different.

The good news: you don't need them. To build a pipeline, AI needs the format of the logs, not their content. The structure and the fields are enough; the real values are not needed. A firewall syslog line with a fake IP and a made-up hostname is just as useful to the pipeline as the real one.

There are two options:

• Obfuscated client data (preferred). Real samples, with every sensitive value replaced. I don't do that by hand but with a script that runs fully offline. AI can write that script for you as well. It replaces IP addresses, hostnames, users and other values with fake ones, and does so consistently: the same host gets the same fake name everywhere, so the data still hangs together.
• Synthetic data. From the log source's official documentation, I have AI generate sample logs in the right format. Ideal when the source isn't connected yet or you don't have access to it yet.

Real data is preferred: it gives you near certainty that your pipeline will also work in production.

Step 3: the first pipeline, in three files

For every pipeline I create three files:

• Sample data: the obfuscated or synthetic logs.
• The pipeline: the functions, in my style and with comments.
• Knowledge items: the lookups and other knowledge the pipeline uses, such as a table that turns event codes into a readable description.

The first pipeline is always the most work. Not because the AI can't do it, but because I keep steering until it looks exactly the way I want. Comments my way. Some functions in, others out. A lookup instead of a long chain of if statements. A pack instead of loose pipelines. Every correction I make improves the next pipeline.

Step 4: testing in Cribl

Then the pipeline goes into Cribl. There I test it against the sample data and check that the output is right: are the fields parsed correctly, do the lookups do what they should, and does the event come out the way the destination expects it?

If something is off, the correction goes back to Claude Code. That way the lesson ends up in memory too, and the next pipeline gets it right straight away. Only when the output is right do I sign the pipeline off. I bundle the pipelines into one pack, so they can be installed and managed together.

Tip: in Cribl you can download the pipeline's output on the sample data as JSON. Send that back to Claude, as a check or to point straight at an error.

Step 5: the next pipeline, using the project's memory

This is where it gets interesting. Claude Code keeps a memory per project. On top of the instructions in CLAUDE.md, Claude records what it learns: every correction I make and every agreement we reach, including during testing, comes back in the next session. And because the first pipeline lives in the same folder, it can always look at it. So you don't have to explain anything again.

The next pipeline starts with one sentence: "Copy the style from the previous pipeline, but build it for source X this time." The AI already knows how I write comments, which functions I do and don't use, how I set up lookups and what a pack looks like. What is left is the work that matters: understanding the format of source X and checking that the output is right. Then this pipeline goes into Cribl for testing as well, and I repeat steps 4 and 5 for every new source.

So every pipeline makes the next one faster. After a few sources I'm mostly reviewing and testing, and hardly building at all.

Step 6: a review agent looks at the whole environment

This step sits outside the workflow above. It is a separate agent: an AI colleague within Claude Code, with its own rules. It doesn't build anything; it looks across the whole process. After five to ten pipelines it puts everything side by side and reviews the logic, not line by line.

That turns up useful things:

• Pre-processing. Steps that come back in every pipeline, like splitting off a syslog header, are better done once up front, in a pre-processing pipeline.
• Merging. Three syslog sources with nearly the same pipeline? They can be combined, with shared functions. They may also fit in the same pack, which is easier to manage.
• Overlap. Fields named differently from one pipeline to the next, or a lookup that exists in two places.

Tip: whoever checks their own work rarely spots their own mistakes. Because the review agent didn't help build the pipelines and follows its own rules, it catches far more.

The result: up to 10 times faster, with better-quality pipelines

Across several clients I went from one or two pipelines a week to ten, sometimes as many as twenty. Some are fully finished and tested; the rest are ready on synthetic data and can go live as soon as the source is connected. Because every pipeline follows the same agreements, they are also easier to review, maintain and hand over.

To be clear: AI doesn't replace the expertise. I decide the style and the architecture, I test every pipeline in Cribl and I sign it off. AI takes over the time-consuming part: building the pipeline itself and writing the functions.

Cribl is going all in on AI too

Cribl itself is leaning more and more on AI. You can now even shape the Cribl interface to your liking, by using AI to build apps that do it for you.

The Cribl Marketplace is full of these apps, and many of them were built with AI too. One app comes very close to my approach: Pack Builder. It uses AI to build packs from your samples, and it can even work in bulk. A good option if you'd rather stay inside Cribl. You'll also find other apps there, for example to download samples.

Get in touch

📬 Want to learn more about this approach, or need help onboarding sources into Cribl? Get in touch via the options below, or send me a message on LinkedIn.