Logging Strategies for Distributed Java Applications in 1999

In 1999 there was no ELK stack, no Splunk, no Datadog. Log4j 1.0 was a few months away. If you wanted logs from a distributed Java application, you either wrote to files on each host and grep'd them manually, or you built something yourself. At Motorola we built something ourselves.

This is what logging looked like for a real Java application before the ecosystem caught up.

The Problem with `System.out.println`

Every Java developer starts with System.out.println. The problems compound as systems grow:

No levels. Debug output and critical errors look identical.
No context. Which thread? Which component? Which host?
No filtering. You either see everything or nothing.
Not thread-safe. On JDK 1.1, concurrent println calls interleaved output.
Synchronous. Every log call blocks until the write completes.

For a single-threaded application these are minor annoyances. For a distributed, multi-threaded network management system, they made logs unusable.

A Minimal Logging Framework

We built a logger before Log4j existed:

public class Logger {
    public enum Level { DEBUG, INFO, WARN, ERROR }

    private static Level    threshold  = Level.INFO;
    private static PrintWriter logFile = null;

    public static void configure(Level level, String filePath) throws IOException {
        threshold = level;
        logFile   = new PrintWriter(
                        new BufferedWriter(
                            new FileWriter(filePath, true /* append */)));
    }

    public static synchronized void log(Level level, String component,
                                        String message, Throwable t) {
        if (level.ordinal() < threshold.ordinal()) return;

        String thread = Thread.currentThread().getName();
        String ts     = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS")
                            .format(new Date());

        String line = String.format("%s [%-5s] [%s] [%s] %s",
            ts, level.name(), component, thread, message);

        if (logFile != null) {
            logFile.println(line);
            if (t != null) t.printStackTrace(logFile);
            logFile.flush();
        } else {
            System.out.println(line);
            if (t != null) t.printStackTrace(System.out);
        }
    }

    // Convenience methods
    public static void debug(String comp, String msg) { log(Level.DEBUG, comp, msg, null); }
    public static void info(String comp, String msg)  { log(Level.INFO,  comp, msg, null); }
    public static void warn(String comp, String msg)  { log(Level.WARN,  comp, msg, null); }
    public static void error(String comp, String msg, Throwable t) {
        log(Level.ERROR, comp, msg, t);
    }
}

Usage:

public class DevicePoller {
    private static final String COMPONENT = "DevicePoller";

    public void poll(String ip) {
        Logger.debug(COMPONENT, "Polling " + ip);
        try {
            DeviceStatus status = snmpClient.get(ip);
            Logger.info(COMPONENT, "Poll complete: " + ip + " → " + status);
        } catch (SnmpException e) {
            Logger.error(COMPONENT, "Poll failed for " + ip, e);
        }
    }
}

Output:

1999-07-14 10:23:41.045 [DEBUG] [DevicePoller] [poller-3] Polling 192.168.1.10
1999-07-14 10:23:41.891 [INFO ] [DevicePoller] [poller-3] Poll complete: 192.168.1.10 → UP
1999-07-14 10:23:45.002 [ERROR] [DevicePoller] [poller-7] Poll failed for 192.168.1.15
com.motorola.nms.SnmpException: Connection timeout
    at com.motorola.nms.SnmpClient.get(SnmpClient.java:82)
    at com.motorola.nms.DevicePoller.poll(DevicePoller.java:34)

Asynchronous Logging

Synchronous logging blocked the calling thread on every write — acceptable for a low-frequency event log, unacceptable for a high-frequency debug log. We made logging asynchronous with a background writer thread:

public class AsyncLogger {
    private final BlockingLogQueue queue      = new BlockingLogQueue(10000);
    private final Thread           writer     = new Thread(this::writeLoop);
    private volatile boolean       running    = true;

    public AsyncLogger(String filePath) throws IOException {
        writer.setDaemon(true);
        writer.start();
    }

    public void log(String message) {
        queue.enqueue(message); // non-blocking: drops message if queue full
    }

    private void writeLoop() {
        while (running || !queue.isEmpty()) {
            String msg = queue.poll(100); // wait up to 100ms
            if (msg != null) {
                writeToFile(msg);
            }
        }
    }
}

The trade-off: log messages could be dropped under extreme load, and messages in the queue were lost on JVM crash. For debugging, acceptable. For audit logs, not.

Centralised Log Aggregation via UDP

With a distributed system on multiple hosts, grep'ing individual log files was painful. We built a log aggregator — a central server that received log entries from all JVMs over UDP:

// Log collector — runs on the central log server
public class LogCollector {
    public void start(int port) throws IOException {
        DatagramSocket socket = new DatagramSocket(port);
        byte[] buf = new byte[4096];
        System.out.println("Log collector listening on :" + port);
        while (true) {
            DatagramPacket pkt = new DatagramPacket(buf, buf.length);
            socket.receive(pkt);
            String entry = new String(pkt.getData(), 0, pkt.getLength(), "UTF-8");
            String source = pkt.getAddress().getHostAddress();
            writeToMasterLog(source, entry);
        }
    }
}

// UDP log appender — added to Logger on each application JVM
public class UdpLogAppender {
    private final DatagramSocket socket;
    private final InetAddress    collectorAddress;
    private final int            collectorPort;

    public UdpLogAppender(String collectorHost, int port) throws Exception {
        this.socket           = new DatagramSocket();
        this.collectorAddress = InetAddress.getByName(collectorHost);
        this.collectorPort    = port;
    }

    public void send(String entry) {
        try {
            byte[]         data = entry.getBytes("UTF-8");
            DatagramPacket pkt  = new DatagramPacket(
                                      data, data.length,
                                      collectorAddress, collectorPort);
            socket.send(pkt);
        } catch (IOException e) {
            // silently drop — logging must not crash the application
        }
    }
}

UDP was appropriate for logging: fire and forget, no acknowledgement, some loss acceptable. The master log on the collector gave us a single place to search across all five NMS JVMs.

What Log4j Gave Us

Log4j 1.0 (January 2000) gave us all of this as a tested, documented library. Hierarchical loggers. Multiple appenders (file, console, socket, syslog) per logger. Configuration without recompiling. Rolling file appenders that automatically rotated logs by size or date.

It took eighteen months for the ecosystem to produce something equivalent to what we had built in eight weeks. This is how most infrastructure software works: a team builds it for a project, it becomes a library, it becomes a standard.