Skip to content
BranchesTags

Repository files navigation

ParseDMARC-go

A Go implementation of the DMARC report parser, based on the original Python parsedmarc project.

📋 Conversion & Enhancements

The conversion to Go was done with Claude AI, adding significant improvements:

✅ Core enhancements:

  • ClickHouse storage with pre-configured Grafana dashboard
  • HTTP reporting method (RUA/RUF with https/http scheme URI)
  • Prometheus daemon mode monitoring (IMAP + HTTP)
  • Directory-based output mode

❌ Not converted (due to lack of testing capability):

  • Elasticsearch/Opensearch/Splunk storage
  • Microsoft Graph and Gmail API support

🌟 Core Features

📊 Report Parsing - Industry leading format support

  • ✅ DMARC Aggregate Reports (RFC 7489)

    • Draft and 1.0 standard formats
    • Compressed file support (GZIP, ZIP)
    • Enhanced error reporting with line numbers

  • ✅ Forensic/Failure Reports (RFC 6591 ARF)

    • Plain text format parsing
    • 🆕 MIME multipart email parsing (LinkedIn, Domain.de, Netease)
    • 🆕 Base64-encoded attachment support
    • Automatic format detection and fallback

  • ✅ SMTP TLS Reports (RFC 8460)

    • Direct JSON format parsing
    • 🆕 MIME email format parsing (Google, other providers)
    • 🆕 Base64 + GZIP compressed attachment pipeline (application/tlsrpt+gzip)
    • Legacy compressed file support (GZIP, ZIP)

🌐 Data Enhancement

  • ✅ IP address geolocation (MaxMind database integration)
  • ✅ Reverse DNS resolution with caching
  • ✅ Base domain extraction and normalization
  • ✅ Enhanced error diagnostics with precise line numbers

📡 Multiple Input Methods

  • ✅ IMAP Email Processing - Monitor mailboxes for incoming reports

    • TLS/SSL connection support
    • Automatic email archiving/deletion
    • Configurable check intervals

  • ✅ HTTP API Server - Receive reports via HTTP POST/PUT (IETF draft)

    • Rate limiting and request validation
    • Multiple content-type support (application/xml, application/json, message/rfc822)
    • File upload size limits and security

💾 Flexible Output & Storage

  • ✅ JSON and CSV output formats with configurable fields
  • ✅ Multiple output modes:
    • File mode: Concatenate all reports in single file
    • 🆕 Directory mode: Save each report as separate timestamped file
    • Stdout: Direct console output for piping
  • ✅ ClickHouse database storage with optimized schema
  • ✅ Email delivery via SMTP with attachment support
  • ✅ Kafka streaming for real-time processing pipelines

📈 Production Monitoring

  • ✅ Built-in Prometheus metrics for observability
  • ✅ Health check endpoints for load balancer integration
  • ✅ Structured logging with configurable levels (JSON/console)
  • ✅ Performance metrics (parsing duration, success/failure rates)

Quick Start

# Download and install
curl -L -o parsedmarc-go https://github.com/CaboneMailServer/ParseDMarc-go/releases/latest/download/parsedmarc-go-linux-amd64
chmod +x parsedmarc-go

# Parse a report
./parsedmarc-go -input report.xml

# Run as daemon
./parsedmarc-go -daemon -config config.yaml

For detailed usage instructions, see the 📖 Documentation below.

🗄️ ClickHouse Database Schema

The program automatically creates optimized production-ready tables with proper indexing, partitioning, and performance optimizations:

📋 dmarc_aggregate_reports

Main aggregate report metadata table

  • Report metadata (organization, report ID, date range)
  • Policy information (DMARC alignment settings)
  • Monthly partitioning by report date
  • Bloom filter indexes on org_name and report_id

📊 dmarc_aggregate_records

Individual aggregate report records

  • Source IP analysis (IP, country, reverse DNS)
  • Authentication results (SPF, DKIM, DMARC alignment)
  • Message counts and policy evaluation results
  • Monthly partitioning with geolocation indexing

🔍 dmarc_forensic_reports

Forensic/failure report details

  • Authentication failure analysis
  • Source information and sample headers
  • Parsed sample message content
  • Indexed by arrival date and source IP

🔐 dmarc_smtp_tls_reports

SMTP TLS report metadata

  • Organization and policy information
  • Success/failure session counts
  • Policy domain and type information
  • Time-based partitioning for performance

⚠️ dmarc_smtp_tls_failures

Detailed SMTP TLS failure analysis

  • Failure types and error codes
  • MTA connection details (sending/receiving IPs)
  • MX hostname and HELO information
  • Normalized for efficient failure pattern analysis

🚀 Performance Features

  • Time-based partitioning: Monthly partitions for optimal query performance
  • Bloom filter indexes: Fast lookups on report IDs and domains
  • Optimized data types: Efficient storage with proper nullable fields
  • Query-optimized structure: Denormalized where appropriate for analytics

🔧 Advanced Email Format Support

parsedmarc-go features industry-leading email format compatibility, automatically handling complex report formats from major email service providers:

🎯 Forensic Reports (RUF) - Universal Compatibility

Plain Text Format

Simple feedback reports embedded directly in email body text

MIME Multipart Email Formats ⭐

Automatically parsed with full provider compatibility:

Provider Format Encoding Content-Type
LinkedIn multipart/report Plain text message/feedback-report
Domain.de multipart/report Plain text message/feedback-report; name=report
Netease multipart/mixed Base64 message/feedback-report; name="ATT00001"
Others Auto-detected Base64/Plain Various MIME types

🚀 Advanced Processing Pipeline:

  1. Multi-line header parsing - Handles wrapped Content-Type headers
  2. MIME boundary extraction - Robust parsing of complex boundaries
  3. Base64 decoding - Automatic detection and decoding
  4. Content-type detection - Intelligent format recognition
  5. Fallback mechanisms - Plain text parsing if MIME fails

📧 SMTP TLS Reports - Next-Generation Support

Direct JSON Format

Standard RFC 8460 JSON reports processed natively

Email-Based Reports ⭐

Advanced multi-stage processing pipeline:

Provider Format Pipeline Content-Type
Google multipart/report Base64 → GZIP → JSON application/tlsrpt+gzip
Others Auto-detected Base64 → Compression → JSON application/tlsrpt+*

🔄 Processing Pipeline:

Email Input → MIME Parse → Base64 Decode → GZIP Decompress → JSON Parse → Structured Data

📋 Supported Standards

parsedmarc-go implements the following email authentication and reporting standards with industry-leading compatibility:

  • RFC 7489 - Domain-based Message Authentication, Reporting, and Conformance (DMARC)

    • Aggregate reports (RUA) with enhanced parsing
    • Policy configuration and validation
    • 🆕 Enhanced error diagnostics with line numbers

  • RFC 6591 - Authentication Failure Reporting Using the Abuse Reporting Format

    • Forensic/failure reports (RUF) with MIME support
    • 🆕 Advanced MIME multipart parsing
    • 🆕 Base64-encoded attachment support

  • RFC 8460 - SMTP TLS Reporting

    • TLS connection and policy reporting with email format support
    • 🆕 Email-based reports with compression support
    • 🆕 Complete ClickHouse schema for analytics

📚 Documentation

📖 Table of Contents

🚀 Getting Started

🗄️ Database and Visualization

🔌 API and Integrations

📖 Technical References

🔗 Quick Links

💡 Tip: Start with the complete documentation for an overview, then check the installation guide to get started quickly.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

🙏 Acknowledgments

📞 Issues

About

parsedmarc-go is a CLI utility for parsing DMARC reports. When used with clickhouse and grafana, it works as a self-hosted open-source alternative to commercial DMARC report processing services such as Agari Brand Protection, Dmarcian, OnDMARC, ProofPoint Email Fraud Defense, and Valimail.

cabonemailserver.github.io/ParseDMarc-go/

Topics

dmarcdmarc-reports

Resources

Readme
Apache-2.0 license

Contributing

Contributing
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages