Logdy

Appearance

Watch out! You're reading a series of articles

Logging Formats: From Chaos to Structure ​

It was 2 AM, production was down, and I was staring at logs that looked something like this:

Error connecting to database
Failed to process payment for user
Connection timeout occurred

Yeah, super helpful, right? 😅 No timestamps, no context, no structure - just a wall of text that told me absolutely nothing about what was actually going wrong. That night made me realize the importance of proper log formatting, and I've been a structured logging evangelist ever since.

Why String Concatenation is Not Your Friend ​

We've all been there. Your first instinct might be to do something like this:

typescript
// Please don't do this 🙈
logger.error("Failed to process payment for user " + userId + " with amount " + amount);

Or in Go:

go
// This isn't much better 😬
log.Printf("Failed to process payment for user %s with amount %f", userId, amount)

Sure, it works... until it doesn't. Try grepping through thousands of these logs to find a specific payment issue, and you'll understand why this approach doesn't scale.

Structured Logging: Your Future Self Will Thank You ​

Here's how we can transform that mess into something actually useful. In TypeScript, using a library like Winston:

typescript
import winston from 'winston';

const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [new winston.transports.Console()]
});

// Now this is much better! 🎉
logger.error('Payment processing failed', {
  userId: 'usr_123',
  amount: 99.99,
  currency: 'USD',
  errorCode: 'PAYMENT_FAILED',
  traceId: 'abc123xyz',
  timestamp: new Date().toISOString()
});

// Output:
// {
//   "level": "error",
//   "message": "Payment processing failed",
//   "userId": "usr_123",
//   "amount": 99.99,
//   "currency": "USD",
//   "errorCode": "PAYMENT_FAILED",
//   "traceId": "abc123xyz",
//   "timestamp": "2024-01-15T14:30:00.000Z"
// }

Or in Go, using the fantastic zerolog:

go
package main

import (
    "github.com/rs/zerolog"
    "github.com/rs/zerolog/log"
)

func processPayment() {
    // This is how we roll in Go! 🚀
    log.Error().
        Str("userId", "usr_123").
        Float64("amount", 99.99).
        Str("currency", "USD").
        Str("errorCode", "PAYMENT_FAILED").
        Str("traceId", "abc123xyz").
        Msg("Payment processing failed")
}

Real Talk: Why Structured Logging Makes Life Better ​

Here's a real scenario from my past life as a payment service developer. We had a mysterious issue where payments were failing for certain users, but only during peak hours. With our old string-based logs, finding the pattern was like looking for a needle in a haystack.

After switching to structured logging, we could run queries like:

bash
# Using jq to analyze JSON logs
cat application.log | jq 'select(.errorCode == "PAYMENT_FAILED" and .amount > 1000)'

Within minutes, we discovered that all failed payments over $1,000 were timing out because of a misconfigured connection pool. The structured format made it trivial to correlate the error patterns with specific conditions.

Pro Tips for Structured Logging ​

  1. Be Consistent with Field Names: Pick a naming convention and stick to it. Is it userId or user_id? Choose one and make it a team standard.
typescript
// Define your log fields as types for consistency
interface LogContext {
  userId: string;          // Not user_id or UserId
  requestId: string;       // Not req_id or reqId
  correlationId: string;   // Not correlation_id
}
  1. Add Request Context Using Middleware: Here's a neat trick in Express.js:
typescript
import { v4 as uuidv4 } from 'uuid';

app.use((req, res, next) => {
  req.context = {
    requestId: uuidv4(),
    startTime: Date.now()
  };
  next();
});

// Later in your route handlers
app.post('/api/payment', (req, res) => {
  logger.info('Processing payment', {
    ...req.context,
    amount: req.body.amount,
    userId: req.user.id
  });
});
  1. Don't Forget About Performance: While structured logging is awesome, remember that creating complex objects for every log entry has a cost. In high-throughput services, consider using sampling or level-based filtering:
typescript
if (logger.isDebugEnabled()) {  
  // Only construct object if we're actually going to log it
  logger.debug('Detailed payment info', {
    ...complexPaymentObject,
    ...expensiveToCalculateMetrics
  });
}

The End Result ​

After implementing structured logging, your logs become a goldmine of information. Instead of dreading those 2 AM incidents, you'll have the tools to quickly understand and resolve issues. Plus, you can build awesome dashboards and alerts based on your log data.

Remember: Future you (or the poor soul who maintains your code next) will either curse or bless you based on how you handle logging. Choose wisely! 🧙‍♂️

P.S. If you're dealing with legacy applications that can't be immediately converted to structured logging, consider using log shippers like Logstash or Fluentd with grok patterns to parse unstructured logs into structured formats. It's not ideal, but it's better than nothing!

Watch out! You're reading a series of articles

Copyright © 2024-present Logdy