Monitoring and logging in large companies is a critical aspect of system management, as it helps track the performance and health of applications. A popular practice among some organizations involves using a common data store, usually a time-series database like Elastic, to manage metrics as logs. This setup, often utilizing the ELK stack, includes tools like Logstash.

In this system, almost everything is treated as a log. API latencies, metrics, responses from external APIs, and database queries are all considered logs. This approach is nice because it allows for easy metric creation, making it straightforward to set up alarms or dashboards on top of these metrics.

However, this is hard with untyped and loosely structured logs. Usually this is where a lot of companies start from. Logs often lack structure and can vary widely in size and field count. The goal was to transition our application servers to Elastic, allowing it to manage all logs while enabling us to build metrics and create dashboards based on them, accessible throughout the company.
During implementation, we encountered issues with Elastic’s handling of mixed data types.

Elastic indexes and data types

Elastic requires consistency in data types for fields with the same key. For example, if the first log’s result is a number, Elastic expects all subsequent logs to adhere to this data type. This becomes problematic when different data sources, such as Cashfree, Razorpay, or Google, return varied data types.

Structuring Logs for Efficient Indexing and Debugging

Every log entry contains two key components: metric data points and debug data points. This distinction allows for both efficient indexing and comprehensive debugging.

Metric Data Points: Structured and Indexed

Metric data points capture well-defined, structured information that can be indexed for querying and analysis. For example, if an API call to Google fails, the log should record essential details such as:

• The API endpoint called

• The user ID associated with the request

• The error code or status

This structured data is stored in a predefined log schema, ensuring consistency across logs and enabling Elasticsearch to efficiently index and query it.

Debug Data Points: Contextual but Unindexed

Debug data points provide additional context but are not meant for indexing. These include verbose details such as the full API response, stack traces, or detailed request payloads. To prevent unnecessary indexing overhead, this information is placed in a dedicated field, such as _debugInfo, which is explicitly excluded from indexing.

Balancing Indexing and Debugging

By structuring logs this way, we achieve:

• Efficient querying: Metrics are indexed for dashboards and analytics.

• Comprehensive debugging: Debug information is available for troubleshooting but doesn’t clutter indexed searches.

• Scalability: Consistent log structures prevent schema conflicts and ensure smooth integration with Elasticsearch.

The logger library seamlessly integrates both metric and debug components, ensuring logs are both queryable and human-readable without unnecessary indexing overhead.

Suggested Logger Interface

In this world, there will be methods which are more targeted and structured to make sure elastic can run with ease.

// elastic expects a clean and consistent type so indexing can work well
export interface MetricData {
    applicationEvent: Uppercase<string>;
    logLevel?: LogLevel;

    correlationId?: string;
    apiName?: string;

    count?: number;
    entityId?: string;
    errorCode?: string;
    queryExecutionTimeInMs?: number;

    // all data relating to debugging, this is not searchable on elasticsearch by design
    _debugInfo?: Record<any, any>;
}

and then there will be a logMetric method which is strongly typed and will log this metric data out into a log stream. This is later picked up by logstash and pushed to elastic.

logMetric(ctx: MetricData): void {
        const logLevel: LogLevel = ctx.logLevel ?? 'info';
        this.logger[logLevel]({
            ...ctx,
        });
    }

Some tips

• Setup a strong correlation ID system so that correlation IDs are neatly cascaded across the system, and also source correlation IDs are respected
• Having an API name (this can be the name of the function of the controller) is very useful to breakdown visualisations for slow queries, server latency breaches etc..

Chefs come back home and cook for their family and friends. They look forward to the cooking on the weekend, the “specials” with great excitement. It is a place where there is love & gratitude. No pressure to deliver award winning recipes, just a good meal with jokes and banter. I think there is something beautiful in that.

A lot of Software engineers seem to have gotten into this mentality where things have almost always become about scale. Scale in terms of users & scale in terms of feature set.

Once in a while it is good to just build for ourselves, to build for friends and family. Something which is personal, predictable, customized. Something which is minimal & clean. Not after money, not after popularity & fame, not after hype cycles. Just for us.

With the state of things in open source and AI, it’s much more easier to build home grown software. Engineers should get back to building things for themselves & friends. There is no guilt in building something which is just being used by a handful of people. And it is a testament to the beauty of the mature structure and ecosystems we have around us.

Inspired by Robert Sloan’s beautifully written post called “An app can be home-cooked meal”

I’m into budgeting and tracking expenses. I think a lot about personal finance, have written my thoughts earlier too. But the thing is that all tracking operates on data. And getting data out of the silo in a structured format is pretty hard.

Account aggregator is supposed to be solving for this. But each vendor in turn again locks data inside their silo and make their app more compelling. Until and unless we build plug and play pipelines we can never really do well in terms of developing personal finance management apps which make sense I think.

I’ve recently started using Actual Budget for budgeting for me and my fiancee. Its a very nifty software which can be self hosted. Now the challenge is to get data from the bank to Actual.

Figuring out how to get data into Actual

Here is the idea. HDFC thankfully gives a CSV export option of their bank statement. If I can figure out how to get this data into actual using a library like ActualPy, then everything is sorted.

Note that I’m using HDFC account, based on your bank these details can change (or not!)

Parsing the CSV file

So step one is CSV parsing. Some catches during the process.

• Some lines in the CSV file have more than the required 7 columns. usually because the narration has a comma in it.
• Some times, the ref number / UTR field will have zero. A lot of software relies on a unqiue identifier. In this case, its better to combine narration, date and hope it stays unique. There is technically a possibility it won’t though.

There is one catch, Actual works heavily on payee information.

Parsing Payee Information from the narration

What does this mean? Here is a sample narration from the bank statement

UPI-ZOMATO LTD-ZOMATO-ORDER@PAYTM-PYTM0123456-401537805904-ZOMATO PAYMENT                                                

This seems gibberish, but this is loaded with information. Delimited by `-`, the first segment is the mode of the payment. the second is the payee name, third is the UPI ID, fourth is the IFSC, fifth is something I did not really get. I’m guessing the last is the note sent to the UPI network.

There seems to be a catch. Sometimes to fit in information, the payee name is truncated. Not really sure in what exact circumstances the bank takes a call to truncate the payee name. Need to look at more data and figure that out. Similarly, NEFT & RTGS also follow a similar structure. So, parsing the payee information is actually fairly convenient.

With this, more than 80% of the transactions would be covered. But then there are other important ones

ACH narrations, ACH stands for automated clearing house. If you happen to have a recurring eNACH, or a mutual fund has to post returns back to your account you’ll incur a ACH line item in your bank statement. ACH narrations are interesting because I’ve seen multiple variations in my bank statement itself. Usually it is ACH followed by a C if its a credit or D if its a debit. then after a hyphen, there is text explaining the transaction followed by an ID of sorts (I guess?). Need to look into this more

Here are some examples

Currently I’m not setting a payee for these transactions in my personal setup.

Finally, The Actual Budget Importer

This is a streamlit app currently hosted in my scripts repo. It takes in csv bank statement of HDFC & posts data to your actual server.

A simple, no BS application which takes in a csv file, the account you are targeting in the server and just posts transactions.

It takes care of de duplication, payee parsing and other quirks regarding commas in narration etc..

With this, data in actual is finally making sense. Now budgeting on top is going to be simple finally!

Streamlit is a framework in Python meant to turn data scripts into web apps in minutes. If you think about it, many apps are fundamentally data scripts first and web apps next. There are many scripts I have which would work better as simple web UIs.

I’ve been exploring how to get from idea to deployment quickly for a while. Of course, the answer is going to be slightly nuanced, but this approach works well for many simple and/or internal apps.

I recently launched CleanMail, which was actually a simple CLI tool back in 2019 called mail-sanitizer. However, it was using Google API credentials and was fairly inaccessible. Now I’ve moved it into a Streamlit app and deployed it on my Coolify instance, and people are using it frequently. I’ve also started migrating my scripts over to Streamlit since it’ll be easier to manage.

Here are some lessons I’ve learned while developing apps in Streamlit:

Embrace Forms

State management is hard, especially when there are no frameworks to rely on (like Redux, etc.). There is a lot of temptation to have independent elements in the UI and to manage state by hand. This will almost always end up being a bad idea.

Forms also control re-renders, and manual state management usually ends up being a pain in terms of the dreaded “Running” state, which kills the UX for the end user.

Minimize State to Manage

Forms help with this, but since there isn’t a lot of tooling around Streamlit right now, it’s important to minimize potential data present in state. Managing state and especially dependent variables in state is very prone to bugs.

Make Use of Fragments

Re-renders are time-consuming and sometimes irritating. Making use of the st.fragments API means that we can control the effects of state with more granularity. This becomes very important as the app scales up in logic.

If you haven’t used the st.fragments API, please do read about it. It’s quite handy.

LLMs for UI

Once all the core logic is in place, asking Aider to come up with an initial draft of UI is extremely useful. The first draft is surprisingly good and almost always close to what you would want in the end version.

Missing Components in the Ecosystem

Streamlit is awesome for a particular set of apps, but there are some critical components missing in my opinion. These are ordered by importance in my perspective

1. Library for easily installing analytics like plausible, clarity etc..
2. Libraries to simplify state management

Apple intelligence is slowly rolling out, meanwhile people are building their own versions of voice assistants similar to Alexa & Siri. I think its a good time to think about what prompts we can expect to work in a basic voice assistant, this might serve as a good test case repository to see how voice assistants perform.

This is excluding very basic prompts like

• set a timer for 15 minutes
• remind me to switch off the geyser after fifteen minutes

and many more. These are already well supported by most of the voice assistants out there.

Prompts

• Set an alarm three hours before the flight departure tomorrow
  • Should read the calendar, figure out that a meeting invite for a flight is set to 9am in the morning. Set an alarm at 6AM.
• Mark schedule visit to dentist as complete
  • Should query reminders, find out what’s the reminder about the dentist and mark it complete

Speed typing is the idea of working and improving on typing speed. It is said that an average person types around 40 words per minute (or WPM for short).

The reason I want to type fast is because I write a lot of blogs/notes. I don’t mind programming with a slower WPM since anyways that’s not the bottleneck. But during blogging, I would prefer if I can type at the pace of my thoughts.

People say typing speed does not matter in programming. I think that is not entirely true, if you are above 60 wpm I think you are good. If not, typing should not be a hindrance to think and iterate for programming is what my I think.

The journey to 100 WPM

My current baseline is around 50-70 WPM, depending on the tool. On KeyBR, I was told “Your all time average speed beats 50.19% of all other people. Average speed: 51.7 wpm”

Now on how to improve from here

Tools to practice with

There are a couple of categories here

Games

The most fun ones

• ZType: This blew my mind. You basically type words and on a word press, one enemy ship goes down. So basically words are ammo. 🤯
• TypeRacer: Fun game to play against friends

Practice tools

• MonkeyType: My current favorite. Great UI, lot of customization. Highly recommended.
• KeyBR: Ruthless, long winded. probably the most useful since it says which keys I suffer with.

Heads up: Heavily WIP. Still figuring it out.

• Sleep is a lot more important than people think. Make sure you are getting a solid 7 hours of sleep every day.
• Get a full body health checkup done start of every year. Get a doctor consultation to figure out an interpretation of readings and schedule top up tests depending on deficiencies.
• Go get a routine ENT & dental checkup every year
• Exercise at least for 2% every day, that is around 28.8 minutes.

Some things I want to try

• VO2 max test

Here is the gist of how all this works together

• Home assistant needs to run on the local network accessible so that devices can ping and communicate with it
• Homekit integration will be using multicast DNS over 21063 (mDNS for short) to establish a homekit bridge between the server on the network & iPhone.
• Since both Tipi & HA run in docker, we’ll need to install mDNS repeater on Tipi and use it to forward mDNS packets from HA.

It might look like a lot, but I promise it’ll be pretty straightforward. Feel free to contact me on twitter if you run into any issues!

Step 1: Configuring Home Assistant

First, we need to make sure Home Assistant is properly configured to work with HomeKit:

1. Open your Home Assistant configuration file (configuration.yml this is situated at runtipi/app-data/homeassistant-1/data/config/configuration.yaml).
2. Add the following HomeKit configuration:

homekit:
  name: Bridge
  port: 21063
  advertise_ip: "192.168.0.101"  # Replace with your Home Assistant IP

3. Save the file and restart Home Assistant.

Step 2: Setting Up mDNS Repeater

To ensure proper communication between HomeKit and Home Assistant, we need to set up mDNS repeater:

1. Identify your network interfaces by running docker exec <container_id> ifconfig , here the container ID is the ID of the docker container of Home assistant
2. run route -n to figure out what’s the network name of the IP address you got above
3. Install mDNS repeater app from runtipi appstore. Start the mDNS repeater with mDNS repeater app with the host interface (e.g., enp1s0) and the Docker network interface (e.g., br-fd25fefeed1f).

Step 3: Configuring RunTipi

Now, let’s configure RunTipi to expose the necessary ports:

1. Navigate to your RunTipi configuration directory: ~/runtipi/user-config/homeassistant-1/
2. Edit the docker-compose.yml file:

services:
  homeassistant-1:
    dns:
      - ${DNS_IP}
    ports:
      - 21063:21063

3. Save the file and restart the Home Assistant app on RunTipi.

Step 4: Setting Up HomeKit Integration

1. In Home Assistant, go to Configuration > Integrations.
2. Add a new integration and search for “HomeKit”.
3. Follow the prompts to set up the HomeKit integration.
4. Once complete, you’ll see a QR code or a pairing code.

Step 5: Pairing with iOS Device

1. Open the Home app on your iOS device.
2. Tap the “+” button to add a new accessory.
3. Scan the QR code or manually enter the pairing code provided by Home Assistant.
4. Wait for the connection to be established. This may take a moment.

Troubleshooting

If you encounter issues:

1. Ensure all ports are correctly forwarded and not blocked by firewalls.
2. Double-check that the mDNS repeater is running correctly.
3. Verify that the HomeKit integration in Home Assistant is using the correct port (21063).
4. If using Tailscale or other VPN solutions, ensure they’re not interfering with local network discovery.

Conclusion

By following these steps, you should now have Home Assistant successfully integrated with HomeKit through RunTipi. This setup allows you to control your smart home devices using Apple’s Home app and Siri, providing a seamless experience across your Apple devices.

Special mention & Massive shout-out to jigsawfr from Runtipi discord for helping me set it up for the first time!

Initially software was bought and sold in CDs. If you bought a copy of windows 7, you’ll get a windows key along with the CD. And you could use the key along with the CD and you owned the software. No recurring subscriptions etc.. Later on if Windows 8 comes up, you would have to go and get the upgrade.

The reason this is great is because there is one simple predictable pricing structure & clear ownership.

This is fundamentally different from leasing software with subscriptions. Any software without significant recurring cost should IMO offer a single time license. They can always say that the upgrade needs to paid for. But the version of the software you bought is with yours. Even if the company dies.

This is a list of software vendors who support this kind of payment options. They either offer a way for me to buy software and own that version of it or do a lifetime purchase for the service itself.

Note to self: maybe make this a website later on?

Pushover

Description: Send notifications to your devices with web hooks. Basically hosted ntfy/gotify service.

Plan: One purchase per device/platform priced at 5$ last I checked. Lifetime access. A steal if you ask me.

Alfred

Description: The Quintessential quick launcher for Mac.

Plan: One purchase per version.

MixPost

Description: Cross platform social media manager

Plan: One time purchase of 299$ for the pro. lifetime license.

Consider the following use cases, and a hypothetical scenario of what would happen normally in an organization which does not solve these explicitly

1. A developer wants to convert sensitive doc from base64 to plain text
   • people go to public websites & paste PII data due to ignorance
2. People want to shorten links and create bookmarks. for example company.links/mis should take you to the data analytics platform
   • everyone creates there own set of bookmarks, and migrations to new domains are essentially messages on public channels etc..
3. Dashboard of all internal services
   • bookmarks again
4. People want to chat using ChatGPT / Claude
   • they user either personal accounts with their own configuration
5. Marketing wants to transcribe a un-released, probably sensitive video and extract text
   • upload to some website which offers a free tier
6. Convert & Join PDFs
   • some random online tools
7. Share files between devices
   • personal whatsapp or telegram would be used

There are many other use cases where employees default to their own preferred method without really thinking about data security

Instead if the company opts to run a small server under a VPN, then for each of these problems, there is a nice solution which is data sec friendly

Here is how it would look like in a world where the company uses something like runTipi to self host inside a VPN

1. A developer wants to convert sensitive doc from base64 to plain text
   • Cyberchef
2. People want to shorten links and create bookmarks. for example company.links/mis should take you to the data analytics platform
   • Shlink
3. Dashboard of all internal services
   • a dashboard like Dashy or Homepage
4. People want to chat using ChatGPT / Claude
   • Open webUI
5. Marketing wants to transcribe a un-released, probably sensitive video and extract text
   • self hosted whisper based transcriber
6. Convert & Join PDFs
   • Sterling PDFs
7. Share files between devices
   • Something along the lines of Send

The organization can have its own universe of self hosted & well designed tooling which solves these problems really well.

More importantly, self hosting is deemed to be too scary. I disagree. Its actually very much doable with the current compute speeds and docker based backups.

Companies should give it a honest shot at a small scale to start with and see how it goes.

How much time does it take for you to go from imagining a decent idea to having a functional bare bones proof of concept web app?

Right now, for me it’s around two days of intermittent work. Here’s why

My current stack is

• Next js for the back end and SSR
• mongodb as the database
• Tailwind as the css framework
• Firebase as the auth provider
• Prisma as the orm most of the time (or mongoose if I’m with mongodb)
• Some kind of redux alternative if needed

And the list goes on.

I waste the most amount of time on

• Authentication: the firebase/ auth0 integrations of the world
• Deciding on the db
• Deciding on the front end framework
• Spending a stupid amount of time on styling and css libraries

This is pretty disappointing. Idea to go live should be as soon as possible so that iteration speed is great. What’s the point of having insane tech at our disposal if we still struggle to do basic crud apps slowly?

How to move faster on side projects?

There are three common answers to this from the community I hear

1. Php and friends: larvel seems to be pretty good
2. Ruby on Rails : one of the most beloved frameworks for developing web apps. Apparently once you get it, there is no going out.
3. Django: aimed at perfectionists with deadlines. Delivers what it promises absolutely.

I have a (unhealthy) habit of jumping into rabbit holes and spending days or sometimes months obsessing over a topic which is entirely unrelated to my day to day & probably is useless. But, its pretty damn good fun.

So, here are the rabbit holes I jumped into

• Self hosting
• Watches
• Self quantification
• Cardistry
• OneBag

A bit of a story, I recently wanted to buy a mini pc for my home server. That’s a 20+k investment, even if I buy it in second hand. While taking to a friend, I realized that one more common friend had a pc lying around which was unused. After asking him about it, he quickly offered to give it to me since it was lying unused.

A couple of interesting things happened here. I saved a significant amount of money and time, which would have gone into researching and purchasing the pc. My friend got rid of some old unused hardware which was lying around. We effectively contained the carbon footprint of the community too since I did not purchase yet one another pc. I potentially prevented a computer from entering the landfill. That’s a pretty damn good outcome!

On communities

We are social animals, we thrive in communities/tribes. Each and every one of us is potentially part of multiple communities. Family community, workplace community, your friends who you play football with etc..

There are resources scattered in these communities which are wildly under utilised..

Economics of scale is a very powerful fact of life. With volume, almost always price comes down. If communities are formed, a lot of things organically start working out.

Use cases

• Sharing subscriptions like iCloud, google one, YouTube, prime video etc..
• Car pooling across groups like apartment groups & office groups
• Collation of community media and resources like videos, music etc.. For example, uploading pics of an event, sharing videos after an event. Uploading performances for everyone in the community to check out etc..
• Communication like posting events & updates, requesting referrals etc.
• Listings for leasing for or for not money and managing exchanges. For example, I have a raspberry pi at home which I’m not using which I’ll happily lend to a junior at office if he wants to experiment on IoT.

Types

Communities in my opinion are of multiple types by the nature of entry

• By property: you’ll be a part of a community by property of being a part of a family. Or by property of being employed at a place.
• By trust: you and your friends will be a part of the community

Dynamics are quite different in these communities. For example, you might comfortably give away your bike to a friend for a week, but will think twice to give the same bike to a coworker in your office.

Some financially profitable use cases to experiment with

• Backups: backblaze seems to offer a terabyte worth of data storage on hot s3 compatible medium for around 7 dollars a month. For one person, this might be overkill, but for a family of four it makes sense.
• YouTube and iCloud has family plans which could be shared and are fairly cheap per member.

Closeknit: an experiment

Looks like discovery is one of the most important bottleneck. Closeknit aims to be a discovery and management platform for items and subscriptions.

With Closeknit, you can join communities and discover under utilized resources.

Let’s see how this goes.