Topics

Audit Logs, Part 3: Putting Web Service Logging Design into Practice

  • column

On the application side, what should be recorded, and how far should it go?

Hello, I'm Otsuka, CTO at Liberogic.

Last time, we compared the logging features of each service, including AWS, Cloudflare, Vercel, Supabase, and headless CMSs.

Each service provides execution logs, error logs, and records of administrative operations.

So, if you combine them, will you have all the logs a web service needs?

Unfortunately, it is not that simple.

The logs that cloud services record automatically and the logs you must record yourself in the application you build are separate things.

This time, we will walk through how to approach logging design for a real web service, focusing on practical considerations.

What service-side logs alone can't tell you

If you use Cloudflare Workers or Vercel Functions, execution results and errors are recorded automatically to a certain extent.

With Supabase, you can review logs for databases, authentication, APIs, and more.

However, service-side logs only capture what happens within that service.

What tends to be logged automatically

What needs to be logged in the app

Worker and Function executions

Contact form submissions

Exceptions and processing errors

Changes to member information

API responses

Permission changes

Database connections

Success or failure of email sending

Cloud environment management operations

Spam detection and business-related rejections

For example, even if you know that a database write succeeded, you cannot tell whether it was "an inquiry was received" or "member information was changed."

Events like these business-level occurrences need to be recorded with meaning attached by the application itself.

Not everything should be logged

There are times when you may want to log as much information as possible in case you need to investigate later.

However, more logs are not always better.

Recording every single process in detail buries the information you actually need. As the volume of logs grows, so do the costs of storing and searching them.

That is why we design logs with the "boundaries" of each process in mind.

For example, in the case of a contact form,

  • submission was accepted successfully
  • submission was rejected due to invalid input
  • submission was rejected by spam detection
  • saving to the database failed
  • sending the notification email failed

We record points like these, where the outcome of a process changes.

Rather than keeping every minor variable and process, the idea is to select only the events needed to trace what happened later.

Keep logs as data, not as prose

Rather than simply writing "An error occurred," it is more useful to include information that can be searched later.

Item

Role

Timestamp

Confirm when it occurred

Event name

Classify which process it occurred in

Severity

Separate success, warning, and failure

Target ID

Identify the target data

Request ID

Track a series of processes

Processing time

Check for delays and performance degradation

Logs that record these items in a consistent, predefined format are called structured logs.

When items are organized in a format such as JSON, it becomes easier to search for things like "check only the processing related to a specific application ID" or "extract only the failed email deliveries."

Do not write personal information to logs

When logging contact form activity, there are cases where the name, email address, and message body are all recorded.

While this may seem useful for investigations, it is a design choice best avoided.

Write to log

Do not write to log

Processed at

Password

Event name

Access token

Success / Failure

Cookie

Target data ID

Inquiry body

Request ID

Credit card information

Processing time

Unnecessary personal information

Logs may be forwarded to multiple services or kept for long periods. If they contain personal information or authentication credentials, the logs themselves can become a new source of data leaks.

Business data, such as inquiry details, should be stored in the database. The log should only record an ID that identifies the inquiry and the fact that it was received successfully.

When needed, use that ID to match the records against the information in the database.

Thinking Through a Contact Form

Contact forms often use a setup that simply sends a notification email.

However, even when the email is sent successfully, there is no guarantee that it will reach the recipient.

For that reason, it is safer to save the inquiry to the database first and treat the email as a notification to the person in charge.

With this setup, you can tell whether the inquiry never arrived at all or whether it was saved but only the notification email failed.

These small differences matter a great deal in actual inquiry handling.

Separate short-term search from long-term storage

For routine troubleshooting, it is important to be able to quickly search recent logs.

On the other hand, logs kept for audits and incident investigations are rarely viewed day to day, but they need to be retrievable months or even years later.

Storage location

Primary purpose

Each service's log screen

Recent operational checks

Log management services such as Datadog

Cross-service search, monitoring, and alerting

R2 or S3

Long-term evidence retention

Keep the data needed for recent investigations in easily searchable locations, and move long-term evidence to lower-cost storage.

It may seem simple, but this separation makes a real difference in practice.

Logs and dashboards are also separate things.

You might think that with logs alone, you can see everything, including access counts and error rates.

However, the numbers you see in logs and in dashboards serve slightly different purposes.

  • Logs: Examine individual events in detail
  • Metrics: Aggregate counts, ratios, processing times, and similar values
  • Dashboards: Get an overview of overall system trends

Use dashboards for routine status checks, and when anomalies appear, investigate them in detail through logs.

Combining these two makes operations much easier to manage.

Minimum items to decide

Finally, here is a summary of the items to confirm in your logging design.

  • Events to record
  • Fields to include in logs
  • Exclusion of personal and authentication information
  • Period for searching recent logs
  • Scope of logs to retain long-term
  • Storage location for long-term logs
  • Staff authorized to view logs
  • Method for deleting logs past their retention period

When you hear the term audit log, you might feel that every operation must be recorded and that a large-scale log management infrastructure has to be introduced.

On the other hand, for a typical website or media site, you can start by first deciding which important operations and processing results to record, and then separating short-term search from long-term storage.

What matters is not listing the logging features of each service you use, but checking whether the records the system as a whole needs are actually being kept.

What should be kept?

What is the purpose of keeping it?

How long does it need to be kept?

If you organize these three questions, you can avoid an unnecessarily large architecture and more easily design logging that fits the project.

Logs are usually not very noticeable.

However, whether you can explain "what was happening" when something goes wrong is determined by the initial design.

Building this kind of grounded foundation properly is really what matters.

Well then.

About the author

The backbone of Liberogic's technical division. Whenever he hears someone say, "I wish we had something like this, it would be so handy," he adds his own ingenuity, turns the idea into added value, and gets it implemented in no time. A highly sociable team member who has many fans among our clients, he is a treasured asset to our company and a devoted cat lover.

Sho Otsuka

Director, CTO / Chief Engineer / Representative, Nekoana LLC / Looks unreasonably young

View this staff member's articles

Reliable team structure and responsive support are our strengths

Liberogic's experienced staff actively drive projects forward, which has earned high praise from our clients.
We make sure to properly assign project managers and directors to keep the entire project running smoothly. Our style of allocating resources to the right people, rather than adding full-time commitment unnecessarily, helps prevent cost increases. We are also known for our speed, from understanding the scope of work to preparing and submitting estimates.

Note: Please be advised that we do not actively engage in SES-style on-site staffing assignments.

We support nearly all major project management and chat tools, including Slack, Teams, Redmine, Backlog, Asana, Jira, Notion, Google Workspace, Zoom, and Webex.

Have a web challenge? Let us know.

Case Studies