Structred logs

[vikramsubramanian]

Hey ! 👋 🙂

I've been through quite some "issues" with a similar topic (including which I have also participated in! ) but I thought perhaps a new thread is worth it, instead of reviving a closed one. 😅

hashtag Logging Lists & Dicts

Previously, you helped me get to [this]( point:

import sys

from loguru import logger

def formatter(record):
base_format = "{time} {level} {name} {message} " + " " * 10
base = base_format.format_map(record)
lines = str(record["extra"].get("data", "")).splitlines()
indent = "\n" + " " * len(base)
reformatted = base + indent.join(lines)
record["extra"]["reformatted"] = reformatted
return "{extra[reformatted]}\n{exception}"

logger.remove()
logger.add(sys.stderr, format=formatter)

data = """---------------- Request ----------------
Headers : {"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
"Content-Length": "20",
"User-Agent": "python-requests/2.27.1",
"cookie": "foo=bar; bar=baz",
"x-pretty-print": "2"}
URL :
Method : POST
Body : {"foo": "bar"}"""

logger.info("Default message")
logger.bind(data=data).info("Message with data")
However, in the above example, I was logging a str, and instead, I'd like to log a dict:

{
"Headers": {
"Accept": "/",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
"Content-Length": "20",
"User-Agent": "python-requests/2.27.1",
"cookie": "foo=bar; bar=baz",
"x-pretty-print": "2",
},
"URL": "
"Method": "POST",
"Body": {
"foo": "bar",
},
}
Is there a newer, better way of the above solution? Have things changed perhaps? 👀
How could I make it so that I can "auto-format" the output for lists or dicts?
hashtag Logging with indents

Wrt OP's original query, I'd like to also have indentation, but slightly different.

This was the proposed solution:

logger_indentation = ContextVar("logger_indentation", default=0)

def indent_logs(indent_size):
val = logger_indentation.get()
logger_indentation.set(val + indent_size)
yield
logger_indentation.set(val)

def patcher(record):
indentation = logger_indentation.get()
record.update(message=" " * indentation + record["message"])
And in my use-case, I am doing this:

logger.info(f"Parameters:")
for param_info in param_info_list:
with indent_logs(4):
logger.info(param_info)
With output:

2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:129 - Parameters:
2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:132 - element = "some_str"
2023-12-04 15:31:27.058 | INFO | actions_wrapper:action_wrapper:132 - element_index = 0 (Default)
But, what I'd preferably like to have is one log statement:

2023-12-04 15:31:27.057 | INFO | actions_wrapper:action_wrapper:129 - Parameters:
element = "some_str"
element_index = 0 (Default)
And preferably not having to use a context manager - maybe something similar to using bind()? 🤔

[mayil-ai]

To improve the logging format for structured data such as dictionaries and lists, follow these steps:

1. Define a custom formatter function that checks if the message is a dictionary or list and formats it with indentation:

import json

def custom_formatter(record):
    message = record["message"]
    if isinstance(message, (dict, list)):
        record["message"] = json.dumps(message, indent=4)
    else:
        record["message"] = str(message)
    return "{time} {level} {message}\n{exception}".format(**record)

2. When logging structured data, use the bind() method to add it to the extra field of the log record:

logger.bind(structured_data=my_dict_or_list)

3. If you need to modify the log record further, use the patch() method:

def patcher(record):
    # Your modifications to the record go here
    pass

logger.patch(patcher)

4. Apply the custom formatter to your logger:

logger.add(sys.stderr, format=custom_formatter)

By implementing these steps, you will be able to log structured data with better readability, as dictionaries and lists will be pretty-printed with indentation. The bind() method attaches the structured data to the log record, and the patch() method allows for additional modifications before the record is processed by handlers.

Code snippets to check

• loguru/_logger.py:1385-2086 | The 'bind' and 'contextualize' methods are relevant for adding custom context and improving structured data logging.

• loguru/_handler.py:212-288 | The '_serialize_record' function is crucial for formatting structured data like dictionaries for logging.

DEBUG: Run completed