Logpush Transformers let you filter and reshape Cloudflare logs

Cloudflare has made Logpush available on all self-serve plans and declared Transformers generally available, according to the Cloudflare Blog. Logpush Transformers let you write a SQL query to filter, redact, and reshape logs before they are exported, so destinations receive only the fields and records they need.

Logpush Transformers in your export pipeline

According to the Cloudflare Blog, Logpush exports logs in batches and Transformers run inside the Logpush export workflow. A Transformer is a SQL query that references fields from a Logpush dataset. Logpush applies the query to each batch, removing records that fail the WHERE clause and using the SELECT output as the exported record. The transformed output is exported in NDJSON format.

Cloudflare gives customers a dashboard tool called Transformer Studio and an API for creating, previewing, and managing Transformers. After saving a Transformer, you attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version so teams can review how a Transformer changed over time, the blog says.

What a Transformer can do, with an example

The Cloudflare Blog includes a concrete example that shows common uses. A Transformer can rename and nest fields, redact part of an IP address, add a static provider field to normalize multi-CDN logs, and remove health-check or internal traffic before export. Cloudflare shows a SQL SELECT that converts timestamps to unix seconds, masks parts of ClientIP, uppercases HTTP methods, and creates a named request struct, then filters out known internal hosts and health endpoints in the WHERE clause.

Transformers do not change serialization formats, the post says: both input and output remain NDJSON. The effect is that the destination receives a normalized, smaller record, which can lower ingestion and storage costs and reduce downstream processing.

How Cloudflare runs transforms at scale

Cloudflare built Transformers to match Logpush operational requirements, the blog explains. Logpush and Bufstream operate at very high throughput: Cloudflare reports ingesting more than one billion events per second, and Bufstream delivers more than 110 million Logpush records per second. To keep transforms reliable and fast, Cloudflare ran an initial prototype using WebAssembly Transformers in Cloudflare Workers, but decided against a remote call in the export path because invoking a Worker would add a network hop and could introduce transient timeouts.

Instead, Cloudflare embedded a SQL engine in-process. They evaluated engines and chose Apache DataFusion, which is a Rust query engine built on Apache Arrow. In a benchmark, running a prepared Transformer on a synthetic 5,000-row http_requests batch from a single Go goroutine took about 3 ms, which the blog equates to roughly 1.6 million rows per second in that test. To connect their Go-based Bufstream delivery service to the Rust DataFusion engine, Cloudflare uses Arrow IPC over a C ABI and cgo, letting Bufstream hand typed Arrow RecordBatches to DataFusion without per-record parsing.

The design keeps ownership and lifetimes explicit across the Go and Rust runtimes: Go passes a pointer to the input Arrow IPC buffer only for the duration of the cgo call, Rust does not retain that pointer, and Rust owns the output buffer which Go copies into Go-managed memory before releasing the Rust allocation. This avoids cross-runtime memory-management bugs while keeping transforms batch-oriented.

What this means for small orgs, nonprofits, and public teams

According to the Cloudflare Blog, attaching a Transformer to a Logpush job moves common preprocessing into Cloudflare's export path. That can reduce the volume of data you pay to export, ingest, and store, and it eliminates the need to run a separate transformation service just to drop health checks or redact sensitive fields.

Practically, you can use Transformers to:

• Normalize fields from multiple CDNs by adding a static provider value before logs reach your analytics platform.
• Remove internal or health-check traffic so you do not pay to export and store irrelevant records.
• Redact or remove sensitive fields before logs leave Cloudflare.
• Reshape records to match schemas your downstream tools expect, reducing ETL work.

What you should do next

• Inventory the Logpush datasets you export and identify fields or records you always drop or must redact. Start with high-volume jobs.
• Build a simple Transformer that filters known health-check hosts and paths, and use Transformer Studio or the API to preview the result on sample batches, as described by Cloudflare.
• Test transforms on a staging or low-volume job, and watch job CPU and delivery metrics because transforms add per-job CPU work and use Bufstream's per-job parallelism for high-volume flows.
• Use the versioning feature for Transformers so you can roll back changes and review historical edits.
• Consider whether normalizing a provider field or reshaping records can let you consolidate downstream pipelines and reduce ingestion costs.
• If you want help designing or testing Transformers for WordPress, Formidable Forms, or other sites, contact our team at /contact.html.

Operational notes to watch for

Cloudflare says that attaching a Transformer adds CPU work and that Bufstream uses per-job transform parallelism so high-volume jobs can execute batches concurrently. Because Transformers run in-process, they avoid a network hop but still require batch serialization and memory handling via Arrow IPC. Test performance on representative batches and use Cloudflare's preview tools before applying changes to critical export jobs.