On the application side, what should be recorded, and how far should it go?
Hello, I'm Otsuka, CTO at Liberogic.
Last time, we compared the logging features of each service, including AWS, Cloudflare, Vercel, Supabase, and headless CMSs.
Each service provides execution logs, error logs, and records of administrative operations.
So, if you combine them, will you have all the logs a web service needs?
Unfortunately, it is not that simple.
The logs that cloud services record automatically and the logs you must record yourself in the application you build are separate things.
This time, we will walk through how to approach logging design for a real web service, focusing on practical considerations.
What service-side logs alone can't tell you
If you use Cloudflare Workers or Vercel Functions, execution results and errors are recorded automatically to a certain extent.
With Supabase, you can review logs for databases, authentication, APIs, and more.
However, service-side logs only capture what happens within that service.
What tends to be logged automatically | What needs to be logged in the app |
|---|---|
Worker and Function executions | Contact form submissions |
Exceptions and processing errors | Changes to member information |
API responses | Permission changes |
Database connections | Success or failure of email sending |
Cloud environment management operations | Spam detection and business-related rejections |
For example, even if you know that a database write succeeded, you cannot tell whether it was "an inquiry was received" or "member information was changed."
Events like these business-level occurrences need to be recorded with meaning attached by the application itself.
Not everything should be logged
There are times when you may want to log as much information as possible in case you need to investigate later.
However, more logs are not always better.
Recording every single process in detail buries the information you actually need. As the volume of logs grows, so do the costs of storing and searching them.
That is why we design logs with the "boundaries" of each process in mind.
For example, in the case of a contact form,
- submission was accepted successfully
- submission was rejected due to invalid input
- submission was rejected by spam detection
- saving to the database failed
- sending the notification email failed
We record points like these, where the outcome of a process changes.
Rather than keeping every minor variable and process, the idea is to select only the events needed to trace what happened later.
Keep logs as data, not as prose
Rather than simply writing "An error occurred," it is more useful to include information that can be searched later.
Item | Role |
|---|---|
Timestamp | Confirm when it occurred |
Event name | Classify which process it occurred in |
Severity | Separate success, warning, and failure |
Target ID | Identify the target data |
Request ID | Track a series of processes |
Processing time | Check for delays and performance degradation |
Logs that record these items in a consistent, predefined format are called structured logs.
When items are organized in a format such as JSON, it becomes easier to search for things like "check only the processing related to a specific application ID" or "extract only the failed email deliveries."
Do not write personal information to logs
When logging contact form activity, there are cases where the name, email address, and message body are all recorded.
While this may seem useful for investigations, it is a design choice best avoided.
Write to log | Do not write to log |
|---|---|
Processed at | Password |
Event name | Access token |
Success / Failure | Cookie |
Target data ID | Inquiry body |
Request ID | Credit card information |
Processing time | Unnecessary personal information |
Logs may be forwarded to multiple services or kept for long periods. If they contain personal information or authentication credentials, the logs themselves can become a new source of data leaks.
Business data, such as inquiry details, should be stored in the database. The log should only record an ID that identifies the inquiry and the fact that it was received successfully.
When needed, use that ID to match the records against the information in the database.
Thinking Through a Contact Form
Contact forms often use a setup that simply sends a notification email.
However, even when the email is sent successfully, there is no guarantee that it will reach the recipient.
For that reason, it is safer to save the inquiry to the database first and treat the email as a notification to the person in charge.
With this setup, you can tell whether the inquiry never arrived at all or whether it was saved but only the notification email failed.
These small differences matter a great deal in actual inquiry handling.
Separate short-term search from long-term storage
For routine troubleshooting, it is important to be able to quickly search recent logs.
On the other hand, logs kept for audits and incident investigations are rarely viewed day to day, but they need to be retrievable months or even years later.
Storage location | Primary purpose |
|---|---|
Each service's log screen | Recent operational checks |
Log management services such as Datadog | Cross-service search, monitoring, and alerting |
R2 or S3 | Long-term evidence retention |
Keep the data needed for recent investigations in easily searchable locations, and move long-term evidence to lower-cost storage.
It may seem simple, but this separation makes a real difference in practice.
Logs and dashboards are also separate things.
You might think that with logs alone, you can see everything, including access counts and error rates.
However, the numbers you see in logs and in dashboards serve slightly different purposes.
- Logs: Examine individual events in detail
- Metrics: Aggregate counts, ratios, processing times, and similar values
- Dashboards: Get an overview of overall system trends
Use dashboards for routine status checks, and when anomalies appear, investigate them in detail through logs.
Combining these two makes operations much easier to manage.
Minimum items to decide
Finally, here is a summary of the items to confirm in your logging design.
- Events to record
- Fields to include in logs
- Exclusion of personal and authentication information
- Period for searching recent logs
- Scope of logs to retain long-term
- Storage location for long-term logs
- Staff authorized to view logs
- Method for deleting logs past their retention period
When you hear the term audit log, you might feel that every operation must be recorded and that a large-scale log management infrastructure has to be introduced.
On the other hand, for a typical website or media site, you can start by first deciding which important operations and processing results to record, and then separating short-term search from long-term storage.
What matters is not listing the logging features of each service you use, but checking whether the records the system as a whole needs are actually being kept.
What should be kept?
What is the purpose of keeping it?
How long does it need to be kept?
If you organize these three questions, you can avoid an unnecessarily large architecture and more easily design logging that fits the project.
Logs are usually not very noticeable.
However, whether you can explain "what was happening" when something goes wrong is determined by the initial design.
Building this kind of grounded foundation properly is really what matters.
Well then.
The backbone of Liberogic's technical division. Whenever he hears someone say, "I wish we had something like this, it would be so handy," he adds his own ingenuity, turns the idea into added value, and gets it implemented in no time. A highly sociable team member who has many fans among our clients, he is a treasured asset to our company and a devoted cat lover.
Sho Otsuka
Director, CTO / Chief Engineer / Representative, Nekoana LLC / Looks unreasonably young