Note
This project is a Work in Progress (WIP). The application is in active prototyping.
This tool is meant to help diagnose system failures accurately and efficiently -locally- by helping address 3 main structural problems of observability (log analysis):
- Semantic blindness:
Traditional reporting tools are strictly dependent on the search the engineer/technician is conducting. As such, simply put, the person who is investigating the failure would actually need to know very clearly the technical phrasing that the back-end was written into.
For example, if a person is typing "service timeout," but the actual error uses the phrasing: "socket exception: connection is lost," the engineer finds nothing > hit and miss, meaning they fail to find the error logs. This implementation aims at addressing this specific problem.
- Cognitive Overload (Logs analysis)
When a system crashes or even if just a component malfunctions, the engineers investigating the issue are handling a wall of text and error logs, sometimes millions of entries, which can increase the incident resolution time - MTTR (Mean Time to Repair/Recover), and increases the likelihood of human error.
Ideally, the 3rd phase of this system would include an AI - something that acts as an automated RCA (Root-cause analysis). This system would introduce a generative AI model at the RAG pipeline. The AI would provide analytical reasoning for analyzing and finding the issue: extracting the timestamp and the cause of the problem/failure.
- System Sovereignty & Compliance:
Many companies are mindful of a specific problem: compliance and how they handle data. When faced with this challenge, a lot of them choose to build in-house implementations due to the sensitivity of the matter.
Sending system and services' logs out for analysis to public cloud APIs is a huge exposure risk because logs contain sensitive information such as IP addresses, raw paths and data entries, API endpoints, and, more rarely but still possible, even PII (personally identifiable information such as email addresses or device information).
By building everything locally, privacy is strictly maintained, confined to local servers, and everything is managed in-house. When it comes to AI usage, by connecting locally an autonomous Agentic AI model that thinks and helps diagnose system problems, privacy issues are directly addressed without sending data elsewhere for analysis.
Note on Phase 3: Step number 3 is "the cherry on top" and completely optional. The Agentic AI execution layer is entirely modular and optional. Given that autonomous AI orchestration is an evolving field and still experimental, this tool is engineered to function reliably by using a passive RAG pipeline; hence, the application can function with or without the active agent loop enabled.
The project is intended to address real-world operational challenges in logs' analysis, observability, and AI agents/orchestration.
This content and application are intended for learning and practicing idiomatic development and system optimization. I care about improving my skills, innovation, learning, and using AI agents efficiently.