Suraj Nath

My 4G setup for working remotely from anywhere

Sun, 27 Mar 2022 00:00:00 +0000

In this post, I will share details of my 4G setup and show how you can build a similar setup for working from remote locations.

Intro

After getting locked in for almost 2 years due to COVID-19, I wanted to spend time with nature, away from busy city life, and breath fresh air. My parents own a farm, so it felt like a perfect place to spend a few weeks. The only issue is that I don’t have wired internet connectivity in the Farm (kinda obvious when we think about it).

The only way to access the internet is over 4G. We get good signal from Airtel & Jio. On mobile phone we were getting around 10-15 Mbps speed but it was not stable (had high ping (>100ms), and jitter(>50ms) which is not good for video calls). Another issue was that speed would sometimes drop down to 1-3 Mbps.

Good enough for basic browsing but NOT good enough to make video calls, share screen and present something to folks on the other side of the world.

Initial research

I started looking into options and found out about microwave Internet (wireless broadband connection delivered by high capacity microwave radio link). Had a chat with a microwave internet company in the area, and abandoned the plan right after looking at the quote. I needed a microwave tower and expensive radios for this setup.

This was too much money for me, I was not ready to spend this much money to work from a farm a few days a year. I wanted a cheap setup that can provide stable 4G internet for a few weeks.

I started reading up and researching ways to get better signals and stabilize the existing 4G connection. I was look for setup that can provide stable connection with decent ping, and jitter and hopefully better and stable network speeds.

I started looking into how the whole telecom infrastructure works. I came across signal boosting products (which are not legal in India unless installed by network providor for you) for remote areas, I saw videos of folks using external antennas with 4G pocket routers to get better internet in RVs.

Around the same time, someone in the Remote Indian community posted about how they are using a 4G router with a cheap external dipole antenna to work from hills.

That post gave me a direction to research, and I started looking into using a 4G router with external antennas.

I started looking into different antenna types, what they do, how they work, and what different units and abbreviations mean. To be honest, half of the things were hard to understand, I was frantically googling while watching the videos and reading blog posts to understand things.

I am trying to explain things in easy to understand words in this blog post, check out this playlist to go down the rabbit hole

Telecom and Communication Basics

Let’s take a little detour into telecom infrastructure to get an idea of how it all works. Feel free to skip this section, this section is for those who want to know details.

I will keep things short, so please refer to the attached sources for more details

Cellular networks have cell towers (also known as cell-site or telecom base station) that host antennas, mostly mounted at high places.

To find these towers/base stations around you, checkout cellmapper.net or Network Cell Info Lite app.

Later we will use the Network Cell Info Lite app to do the site survey, so install and play around with the app.

Bands and Frequency Range

1, 3, 5, 8, 40 are some of the bands used by networks in India, See this table for more details about cell Network Bands and frequency ranges.

For Indian spectrum allocations, I found https://onlytech.com/india-telecom-spectrum-chart/ helpful and easy to understand.

Low Band number means Lower frequency ranges (850MHz range), and it increases as we go up the band number. Band 40 usually operates in the 2100MHz / 2300MHz range in India.

The actual frequency range depends on the network provider, and the spectrum ranges they are allowed to operate by regulatory bodies.

The only thing we need to know about bands is that lower bands can travel far distances and cover a lot more area but the bandwidth is low.

High band numbers can NOT travel very far compared to low bands but they have high bandwidth.

You can think of bandwidth as pipe, so low bands are narrow pipes that can go to kilometers, and higher bands are wide pipes that don’t go as far compared to narrow pipes.

NOTE: availability of bands and frequency ranges will vary based on your location, and cell service provider. Check details of bands and frequency ranges in your country and area.

Terms and Units of Measurements

We will see some 3 letter abbreviations in mobile network literature and the Network Cell Info Lite app. Don’t worry about understanding what they are, and what they mean, we just need to take readings and don’t need to know the details.

Cell Identifiers:

MCC: Mobile country code
MNC: Mobile Network Code
Band: Frequency range, see more
TAC: Tracking Area Code
ECI: E-UTRAN Cell Identity
PCI: Physical Cell Identifiers

See more: LTE Numbering & Addressing

Signal Quality:

RSRP: Reference Signal Received Power
RSRQ: Reference Signal Received Quality
RSSNR: Reference Signal Signal to Noise Ratio
ASU: Arbitrary Strength Unit
Power: Power of the signal

Units:

dB: decibel, used to measure the power of the signal
dBm: decibel-milliwatts, same as dB but a smaller unit
MHz: megahertz
fW: femtowatt, like Watt but way smaller (1 Watt = 1.0E+15 fW)
dBi: dB(isotropic), The forward gain of an antenna, measured in decibels (dBi)

How do we know if we have good signal?

A good signal will have a low Signal to Noise Ratio (RSSNR/ SINR), and High Power (RSRP / RSRQ)

We can use this table to check if we have a quality signal:

Site Survey

Before selecting antenna and other equipment we need to know the signal quality in our target installation area. We need this data to spec out the antenna, and find the best location/direction to mount our antenna.

To collect data about the quality of signal in target area, we will do a Site Survey.

Don’t worry, it’s quite easy. If I can do this, anyone can do this. Site Survey is just a fancy way of saying “Install an app and roam around to find the spot with the best signal”

Go and download the Network Cell Info Lite app on an Android Phone. This app is free on the Play Store, I don’t own an iOS device so you are on your own if you don’t have an Android device.

After Installing, Open, and give it required permissions.

To start the Site Survey, Open the Network Cell Info Lite app, and go to the target location (the area where you want to mount your antenna).

Move around and take note of signal quality metrics like RSRP, RSRQ, RSSNR, and Bands available.

You can switch to RAW tab in app to see all the cell towers around you, and more details about them.

In the RAW tab, “Serving” indicates the cell tower you are connected to and being served by right now. The serving cell tower will show Cell Identifiers along with signal metrics. Below the Serving cell, you will see other cell towers in the area and their signal strength.

Switch back to the Home tab, try moving around, changing direction, going at height, and finding a spot with the best signal. The place with best signal is the ideal location for mounting your antennas.

Note down signal quality metrics like RSRP, RSRQ, RSSNR, and Bands available at that spot. We will use these values to spec our antenna.

If you have SIM cards from different network providers, repeat the site survey with all the SIM cards to find and select the best network provider for your location.

Here are some screenshots from a site survey:

You want to mount your antenna in the areas that have high RSRP (more than -80 dBm is great), High RSRQ (more than -10 is good), and High RSSNR, a positive value of SNR is considered good, and anything above 10 is great. higher SNR means the signal has low noise, and you have a stable signal.

Note: SNR can change with the weather or due to noise from other signal sources nearby. Mount your antenna in a place that’s isolated from noise. High voltage power lines, electric motors, or generators can be the source of the noise.

Finalizing Equipment

4G Router / Modem

We will need a 4G router/modem that supports two external antennas and has MIMO (Multiple Input Multiple Output) support. Routers with MIMO can use two separate antennas at once to deliver fast speeds.

I used TP-Link Archer MR600 AC1200 in my setup because you can connect two external antennas, and it has MIMO capability. I got it from Amazon India.

TP-Link MR600 has MU-MIMO support, and Carrier Aggregation support.

Make sure your router that has an ethernet port, you want to go wired for a better experience, For low latency applications like video calls, wired connections are better than using wifi.

My router has full gigabit ethernet ports, and I connect my laptop using an ethernet cable.

Any router that has these features will work fine, just make sure to get the right type of connectors and cables for your router.

Here are the full specifications (pdf) of my TP-Link MR600 Router.

Antennas

Two types of antenna can be used as the external antenna in a 4G Router. Omni-directional or directional antenna. You need to pick a type based on your use case.

rule of thumb is:

If your setup is movable and you don’t want to adjust the antenna, you can use an Omnidirectional antenna.
If your setup is fixed, then it’s better to use a directional antenna, they will give better results.

weBoost has a great explanation of these types of antenna, also folks from WilsonPro have another good video on Yagi vs Omni Antennas.

My setup was fixed so I used two directional LPDA Antennas with 12dbi gain, and frequency range for my anteena is 698-960Mhz and 1710-2700 Mhz.

This frequency range is used by LTE networks in India.

The gain was calculated based on the readings in our Site Survey, as per my calculations 12dbi gain antenna was enough for me.

To find gain for your antenna, you need the RSRP value from the location where you will mount your antenna and subtract it from your desired RSRP value and then the difference is the gain you need in your antenna.

(-80) dBm RSRP is considered a great value for RSRP so we can take that as desired RSRP value.

For example, If the site survey shows -94 dBm RSRP, you will need an antenna with 14dBi gain to have -80dBm after using that antenna. (-80) - (-94) = 14 dBi gain antenna is required

In real life, the actual gain might be slightly less than the rated gain due to signal loss in cables.

Here is the spec sheet (pdf) for my LPDA antennas:

I bought two antennas and two 15 meter cables from a supplier (signityrfsolutions) on indiamart.com.

You can find these antennas and cables on Amazon India as well.

Cables and Connectors

I used low-loss LMR 400 Cables to connect my antenna with the router, Low loss cables are desired in this setup to reduce signal loss.

LMR 400 cable is used for low-loss communications (rated: loss of 0.223 dB/meter @ 2.4 GHz). Any low-loss cable is fine for this setup, KMR 400 cable also works fine for this setup.

My antennas have an N-female connector, my cables had N-male on one side, and SMA-male connectors.

My 4G router has SMA-female connections for external antennas.

My cables were already crimped, and connectors were attached. I suggest buying cables with connectors installed.

Setting things up / Mounting and connection

You need to mount the antenna at 90-degree polarity OR you can mount them with a good distance between them (if mounted with the same polarity).

Here is how I mounted my antennas:

Testing

After Mounting, I adjusted the antenna while monitoring signal strength metrics (my router has a WebUI that shows these values)

I rotated the antennas in the yaw axis to find a good direction.

After that I adjusted it in the pitch axis to align better with the cell tower radios.

Once you find a good signal, you can permanently fix the antenna at that location.

After fixing antennas, do speed tests to check internet speed, jitter and ping. You should see a difference in speed, jitter, and ping with external antennas connected.

If your speeds are the same with and without an external antenna, look at the stability of speeds. With this setup, you should see low ping, low jitter, and consistent speeds.

I used speed.cloudflare.com to run the tests, It shows the distribution of measurements instead of a single number.

In a stable connection, the measurement values will be close to each other.

Cost Breakdown

Router: TP-Link MR600, Paid 9000 INR (~118 USD)
Antennas: LPDA 12dBi Gain Directional Antenna (2 units), Paid 2000 INR (~27 USD)
Cable: LMR 400 Cable (2 units, 15 meters), Paid 1800 INR (~24 USD)
Antenna Mounts (optional): I used an old bed frame to mount my antenna for my setup.

You can weld two pipes in T share to dish tv antenna base for a cheap mount. Designing an antenna mount is an exercise for the reader, if you end up designing one, tweet it to me @electron0zero

My total cost for the setup was around 13000 INR (~170 USD)

Final Results and Speed Tests

NOTE: I am connecting my laptop via ethernet cable for a stable connection between laptop and router. go wired for a better experience during video calls.

Internet Speeds - 4G on Phone

Internet Speeds - With 4G Router and Extenral Anteenas

Internet Speeds Tests - Different times

Signal Quality at different times - Router UI

Update (Jul 28, 2022): Patrick Hemmen emailed me about configuring route to I can recommend to configure the router to reduce Bufferbloat and recommened testing Bufferbloat with this tool. if you are following this post, I recommened looking into Bufferbloat as well.

Updatte(July 28, 2022): folks in lobste.rs linked to this amazing and detailed post from Geoffrey Huntley on how he works remotly from his van, check it out.

Closing Note

This setup has been working fine for me, I have used this setup to work 2-3 weeks from remote location.

If you end up building a similar setup, would love to see details and photos of your setup (email me).

Hope this was helpful, and you learned something new today 👋

See discussion on Hacker News and lobste.rs

Protect your domain from spammers and Improve email deliverability

Sun, 17 Oct 2021 00:00:00 +0000

In this post, we will talk about how to protect your domain against email spoofing, and how you can configure your domain to keep your emails out of your user’s spam folder.

We will also learn about BIMI Record and how to create BIMI records to show you brand logo in inboxes.

Imagine this, your customers reach out to you saying that they are not getting password reset emails. After some investigation, you find that you sent the email from your end but they are ending up spam folder.

You scratch your head, thinking about why your emails are going spam? you never spammed your customers.

This can happen to you due to missing SPF and DKIM configuration on your domain. Maybe you are a victim of domain spoofing and that’s why your emails are ending up in spam.

Maybe your emails are not landing in spam yet, but if you don’t have SPF and DKIM configured. it’s possible that they will land in your user’s spam folder.

Other than keeping emails out of spam, you also need to stay in the good books of your mail service providers. mail service providers will lock you out if they think you are spamming people.

Configuring SPF, DKIM and DMARC protects your domain against abuse, secures your emails, and improves email deliverability.

Gmail and other public email service provider uses these records are signals for spam classification. A domain with valid records is less likely to end up in spam because google and other email service provider can use SPF, and DKIM to verify the authenticity of the email.

Let’s talk about all the acronyms, what they are and how they can help you.

What is SPF?

SPF stands for Sender Policy Framework. In simple words, you use SPF to tell everyone the mail servers that are allowed to send an email for your domain.

SPF record is used by receiving mail servers to verify that emails for your domain are coming from a mail server that is allowed to send an email for your domain.

SPF records are defined as a TXT record on your domain

Here we are looking at SPF record for google.com and github.com using dig

$ dig google.com -t TXT +short | grep -i spf
"v=spf1 include:_spf.google.com ~all"

$ dig github.com -t TXT +short | grep -i spf
"v=spf1 ip4:192.30.252.0/22 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:1" "66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ~all"

Why do you need SPF?

With an SPF record in place, spammers can’t spoof your domain and send emails pretending to be you because the mail server used by spammers is not an allowed mail server.

SPF protects you against spammers spoofing your domain and ruining the reputation of your domain.

What is DKIM?

DKIM stands for DomainKeys Identified Mail. It’s a way to sign your emails and protect against spoofing after an email leaves the sender’s server.

DKIM allows you to sign your email with a private key as it leaves sending mail server, and then recipient mail servers can use a public key to verify that the email was not changed in transit.

DKIM is also defined as a DNS record on your domain.

Let’s look at the DKIM record for google.

To see the DKIM record we need to find selector and domain in DKIM-Signature in an email we got from google. To find DKIM-Signature we need to look at raw email (in Gmail, click 3 dots, and then click Show Original in Gmail). In the Original email, look for the DKIM-Signature header.

Here is the DKIM Signature I found in an email I got from Google.

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
    d=accounts.google.com; s=20210112;
    h=mime-version:date:feedback-id:message-id:subject:from:to;
    bh=<omitted>;
    b=<omitted>

In this header, we see d= and s= tags along with few other tags like b, v, and a.

d tag is for the domain, and s tag is for the selector, so our full DNS record that contains DKIM public key would be <s>._domainkey.<d>. In our case, It will be 20210112._domainkey.accounts.google.com

Let’s look at it using dig

$ dig 20210112._domainkey.accounts.google.com -t TXT +short
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5jwZiY255HTBgFdh5lrINkcbwuymEytz4kkP1h5KojAVBwyxU42zN6FUwTAI+pq8H6zt77jUrPqh0IUN7m948pHEP0o6xGzKkhOrKuirrtvAKAm/JuRTfVMkrYzHdGbM6X" "hXJ1BFU79vDlNTAsResvhybbUM8FQz1/trnCdhQRka45ZhUpVi1RWdctnhajNwWMBf/FQ4qBVYNWXVaoZnRZp/AtGuwTVkFLQbGPaSmvLELa644g7yo/eCfCUtZpDEKHsDqcHFFg1VUzlnSChnmdyn4Nh61XVP35G18Szt6YT5iUddMPXX5kSKmZ26HSIs/vFHJe0zDDNLkRF3KSLGewIDAQAB"

Why do you need DKIM?

With SPF you can define which email servers can send emails for you but you are still vulnerable to someone modifying messages in between sender and receiver.

DKIM comes into play when you want to protect your emails during transit and protect against man-in-the-middle attacks on your emails.

What is DMARC?

DMARC stands for Domain-based Message Authentication, Reporting, and Conformance

Now you must be asking, why do you need DMARC when you have SPF and DKIM.

As the full form suggests, it allows a domain owner to define what to do with emails that fail SPF or DKIM checks.

in your DMARC record, you can define an email address to get DMARC reports. DMARC reports are sent by receiving email servers and give you visibility into emails that are failing SPF or DKIM checks. you can use DMARC reports to see who is trying to spoof emails for your domain.

Let’s look at DMARC records for google.com and github.com using dig

$ dig _dmarc.google.com -t TXT +short
"v=DMARC1; p=reject; rua=mailto:mailauth-reports@google.com"

$ dig _dmarc.github.com -t TXT +short
"v=DMARC1; p=reject; pct=100; rua=mailto:dmarc@github.com,mailto:d@rua.agari.com"

What is BIMI?

BIMI (pronounced: Bih-mee) stands for Brand Indicators for Message Identification

https://bimigroup.org/ website says that:

For the brand’s logo to be displayed, the email must pass DMARC authentication checks, ensuring that the organization’s domain has not been impersonated.

BIMI is another tool for you to tell your users that email is coming from the real you, and it’s not spoofed. also, having a logo helps you stand out.

Branding ftw!

Let’s look at BIMI record for bimigroup.org using dig

$ dig default._bimi.bimigroup.org -t TXT +short
"v=BIMI1; l=https://bimigroup.org/bimi-sq.svg; a="

You can also use https://bimigroup.org/bimi-generator/ to look and generate BIMI records.

BIMI is relatively new and is only supported by Yahoo, Google, and Fastmail so far. It’s still being adopted by more companies.

This means that If you have a valid BIMI record on your domain, your users will only see the Brand Image if they use Yahoo, Google, or Fastmail products to view your emails.

Tools

Now that we have learned acronyms of the email world, let’s discuss some tools. You can use these tools to test your domain’s configuration, identify and fix issues.

dmarcanalyzer.com/ - they have tools to test SPF, DKIM, DMRC, and BIMI records
toolbox.googleapps.com/apps/checkmx/ - tool to check your email configuration.
bimigroup.org/bimi-generator/ - tool to check MX, SPF, DMARC, and BIMI record. this tool can also be used to generate BIMI record.

Closing Note

Now you know why you need SPF, DKIM, and DMARC records or maybe a BIMI record if you want to be fancy and show your brand image. You can consult your mail server provider’s documentation on how to add these records for your mail server and domain.

In most cases, adding an SPF record is easy, compared to setting up DKIM, and DMARC records.

Some DNS or CDN providers have wizards to add these records. for example, Cloudflare (image attached) shows you an alert with a wizard when your domain is missing these records.

Those were technical things to improve your deliverability.

Let’s discuss some non-technical things to improve your email deliverability.

Only send essential emails, and don’t spam your users, spamming quickly lands you into spam filters and ruins your reputations, it is very tempting to bombard users, but please don’t. if you spam your users, SPF and DKIM will not save you from ending up in the spam folder.
Always provide an unsubscribe button, and make sure it works. users will mark your emails spam if your unsubscribe button doesn’t work
Don’t resort to dark patterns, stay honest, and don’t use clickbait in email subjects. respect your user’s inbox.
Don’t focus on shallow metrics, define and focus on core actions that you want your users to do.
Sometimes a good old text email is better than the HTML email that doesn’t render properly on your user’s device.

I hope this was helpful and helped you secure your domains.

That’s all folks, have a good day and stay safe 👋

Notes on Hardware and Silicon Industry

Thu, 14 Jan 2021 00:00:00 +0000

In this post, I will be talking about rapid changes in the silicon and hardware industry and the reasons why I think we will see the next wave of innovation in this space.

These predictions are based on discussions and what I have been reading past few months. see sources at the end of the post for details.

A few months ago (June 2020), I was talking to my friend Vaishali about what is going in the Linux kernel world. She works with the Linux kernel closely and we started talking about excitement for ARM in kernel maintainers and contributors.

Discussion took an interesting turn and we started talking about chiplets, co-located accelerators and future SOC, chips, and startups working in this space.

Discussions mostly revolved around:

NUVIA, also called “Tesla of Silicon”,
Rise of new rack-scale computer companies like Oxide Computer Company,
Rise of accelerators, and use of GPUs as general-purpose accelerators,
GPU companies expanding into purpose-built accelerators,
Hyperscalers making their own chips.
How Apple is going all-in on ARM, along with Microsoft and Amazon (maybe Google as well, nothing public yet)

Apologies for the buzzword soup, I will try to explain things in details in the coming paragraphs 🙂

I am trying to break it down into sections, and explain all the changes we are seeing in this space, we can think of these sections as big moves that are inter-dependent, but essential for the growth of emerging silicon ecosystem.

Rise of ARM

Big players like Apple, Microsoft, Amazon, and Google are investing heavily in the ARM ecosystem. NVIDIA Acquiring ARM for $40 Billion is the big move that indicates the direction of the industry.

Along with that, big Hyperscalers are betting big on ARM. AWS made its public move by acquiring Annapurna Labs(Israeli startup working on chips) for an estimated $350-$400 million in 2015-16. Annapurna Labs was acquired to bring silicon design capabilities in AWS.

As a result, AWS now offers Graviton, and Graviton2 chips which are ARM-based, and available to AWS customers at cheaper rates compared to Intel chips.

I am sure Azure and Google Cloud are also planning to release ARM-based chips, probably designed by in-house teams, few acquisitions, and some industry collaboration.

Abode is investing heavily in porting software to ARM (Lightroom 4.1 runs natively on Apple M1), and we all know about Apple Silicon-based M1 laptops, and how fast they are.

ARM dominates the mobile and IoT market, and now we are seeing that an ARM takeover is happening in Hyperscaler, and PC market.

Windows is getting support for ARM, and Microsoft is joining forces with Qualcomm, and PC makers to ARM-based computers in the market.

Rise of Accelerators

We are seeing a rise of dedicated accelerators and a new breed of workload-specific SOCs.

Chip designers are trying to get around the performance bottlenecks by co-locating data and compute. See more on chiplets in ACM, March Issue.

We are seeing companies like Tesla building in-house chips to power the self-driving capabilities, and benchmarks show that these chips are extremely good at the job.

Google and NVIDIA have built accelerators for machine learning workloads, and we are seeing the rise of ML processors in mobile SoCs

The more the merrier

When you pack warehouses full of computers, you care about things that normal people don’t even think about. Along with raw per-core performance, heat emission, and power consumption are important for hyperscalers.

Hyperscalers used to be the customers of chipmakers, but they realized that having more control over things like heat and power consumption is important.

When you have a warehouse full of computers, lower power consumption, and less heat can save millions in the power bill.

Because of these, and many other reasons we are seeing hyperscalers investing in their own chips.

AWS, Apple Silicon, Google Silicon, and Tesla making their own chips. We see startups like Nuvia (got acquired by Qualcomm), also working on it.

We are seeing the rise of new Chip Makers, and as the saying goes “the more the merrier”. I am certain this will result in innovation in the space.

The Hardware Software Interface & Rise of Open Hardware, and Firmware

We are seeing a new wave of hardware-based exploits. Meltdown and Spectre opened the pandora’s box, and now we are seeing that x86 optimization techniques like speculative execution are being exploited.

Now hardware will be the place where folks have started looking for optimizations, and security issues. We are seeing the financial industry building their own hardware, and publicly talking about it.

Startups like Oxide computer company are planning to de-blot and modernize servers, projects like Open Compute, and the rise of open firmware means that now it’s possible to design your own hardware and get it built. The barrier to entry is lowered, and now you have a choice.

Closing Note

We are seeing a surge in open source tools, open firmware, and open hardware. The market is ripe for disruption, and we are seeing lots of changes in this space. We are seeing a new wave where x86 is being left behind and big market players are betting billions on ARM.

My Predictions

ARM going to be big in the coming years (US Govt is testing the waters)
RISC-V and open firmware will shake up silicon space
Commoditization of fabrication will happen
Many-Core, Low Power, Purpose-built chips will show up.
Programmable Hardware will make a comeback for those performance critical cases (HFT folks are already doing it)
Rise of Open Firmware & Open tooling to design your own SoC (System on a Chip)
Rise of Security Issues in closed hardware and firmware will drive open firmware adoption

Sources

Read them, verify them, and if you find any errors let me know on Twitter :)

Press & Publications

YouTube

Tooling & Tech

Tweets

Learning Git: Basics and Internals

Sat, 20 Jun 2020 00:00:00 +0000

In this post, I am sharing learning resources that I used for getting started with git, and learning more about git internals.

These resources are organized under the basics and Internals sections.

Basics section is for folks who are getting started with git and learn how to use, and why they should use git.

Internals section is for folks who have used git and want to learn how git works, it’s advance usage, architecture, and internals of git.

Each section contains links to blog posts, videos, books, and documentation. There is an overlap between resources, so pick what you like, and learn from a resource that fits your learning style 🙌

Since we are talking about git, it would a crime not to share xkcd on git. Legend has it that is there is a relevant XKCD for almost everything

Git Basics

Follow this section if you are new to git, and just getting started.

What is git?

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

– Git Homepage

To put it in simple words, It’s a tool that’s used in software development to keep track of the history of changes, tho it’s mainly used for text files but it works for most file types.

This allows software developers to collaboratively work on projects, and allows them to time travel(history of changes) and see what, when, and who changed something.

What is GitHub/GitLab/Bitbucket, and how they are different from git? They are online hosting services for git, with other tools that allow software developers to collaborate on software projects.

Getting Started

If you are looking for a quick getting started guide then gitimmersion.com has a great guided tour of git.

If you have a short attention span like me and like to learn from video, watch this highly compressed Git Explained in 100 Seconds video

If you are looking to learn how to use GitHub, then follow Introduction To GitHub lab from folks at GitHub

There is another set of tutorials from Atlassian (makers of Bitbucket) on What is version control, and another git topics

If you prefer classroom-style lecture with a bit more then basic commands then see this Lecture on Version Control (git) from Missing Semester course by MIT

Git Internals

This section contains resources on how git works

Git from the inside out by Mary Rose Cook. If you like to learn from videos then here is same content in video form

If you want to learn about System Design, History, and Architecture, then checkout this chapter on Git in The Architecture of Open Source Applications Book.

Sidenote: The Architecture of Open Source Applications is a great read and gives us a peek inside (well you guessed it) Architecture of Open Source Applications

Git Internals by John Britton of GitHub - CS50 Tech Talk is a great watch if you want a demo/walk-through style guided tour of Git

GOTO 2015 • Deep Dive into Git • Edward Thomson is another great talk that dives deep into git.

If you are looking for a book on git internals then check out Pro Git book

Other than these resources, the official git website also maintains a list of resources on git. check out that if you are still looking for more resources 🧐

Closing Note

I hope these resources were helpful and will help you learn more about git.

If there is something that you think should be included here, tweet me @electron0zero and let me know 🙂

Happy Learning 👋

LGTM: Writing good enough code

Thu, 18 Jun 2020 00:00:00 +0000

This post is a written version of the internal talk I gave on writing good enough code

It is aimed at folks who are writing production code¹ for the first time.

If you have been writing software for a while then you probably know these things, and you are not the target audience for this post 🙂

While doing code reviews with folks writing production code for the first time I observed few common suggestions, and later I gave an internal talk on the subject. This post is to written version of the talk.

Please keep in mind that these are some of the guidelines(not rules) one can keep in mind to write good enough code, and these are not complete in any way

What do I mean by good enough code?

Code which is easy to understand and gets the job done is good enough, code that meets the requirements is good enough, code that we can ship is good enough.

I believe mindful shipping beats perfection, and we can always iterate to make it better in future.

Our goal should be shipping as fast as we can while trying not to accumulate tech or product debt.

No Code

Best code is code you don’t have, code that we don’t write will not have bugs, will not crash or fail, this is the reason why I love PRs that delete more code then they add

“Every line of code you write is time debt: it is another line that has to be debugged, another line that has to be supported, another line that may require a rewrite later, another line that might cause an interaction with a later feature”

– @patio11, source

PS: Not to be confused with no-code movement

DRY

“Do not Repeat Yourself” (DRY) with some WET code is considered a good practice
Keep things DRY, but don’t DRY too much because then it starts getting hard to read and reason about it.
DRY but don’t compromise readability to DRY something, readability takes precedence over DRYing in my books

Further Reading:

Is Your Code DRY or WET?

YAGNI

YAGNI - “You aren’t gonna need it”
Don’t assume you will need it (unless you are sure, and is part is requirements)
Don’t speculate how It would be used in far future and just write code that gets the job done for the current use-case

Single Responsibility

The Single Responsibility Principle (SRP) states that every class or module in a program should have responsibility for a single piece of that program’s functionality.

It means don’t write a method that does multiple unrelated things, for example, logging modules have a method to format dates, and that formatting feature is used all over the place.

And the same goes for methods, don’t write something that does more than two things. If you need to add some feature in method and when you feel it’s doing two things, refactor it

Write for Readability

If you only want to follow one thing, make it this, because readability matters, and the simple reason is that code is read more then it’s modified, and readable code will be easy to reason about, change, or replace when required.

“Programs should be written for people to read, and only incidentally for machines to execute.”

— Structure and Interpretation of Computer Programs

More time will be spent reading the code you write, you and your co-workers will be reading more code then you will write

Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.

— John Woods

Further Reading:

No Deep Nesting

Deeply nested code is hard to read and hard to understand, it becomes very hard when you have to reason about it or debug it (extra hard when you are in middle of an outage), and as we mentioned readability matters, that’s why deep nesting is considered a code smell

A method with multiple nested if condition quickly becomes hard to comprehend, so try to keep nesting under control.

Use Guard Clauses to get rid of nested conditionals, and keep the branching factor low

Happy Path

The happy path is the optimistic path, It’s the path your code will take under normal conditions, It’s scenario featuring no exceptions or error conditions.

For example, validating input in the start and failing fast will allow you to write code which can work with the assumption that data (down the line) it has it valid since it was validated at the start.

It makes code more readable and easy to debug, also less branching because now you don’t need to handle invalid data

Fail Fast, Fail Early and Fail With Noise

Write your software such that, when there is a problem, it should fail as soon as it detects the problem and knows it can’t recover from it.

Failure should be visible to operator, and failing early is better than trying to proceed in a possibly unstable state where bad state trickles down.

Silent failures are one of the hardest kind of failure to debug, trace and fix, these failures leave no traces for operators, and leave people banging their heads against the wall

You should always leave a visible trace of your failure, raising an exception or logging based on the type of failures

Further Reading:

Shameless plug: I curate failuremodes.dev and failure modes newsletter, do check it out if you interested in how and why software systems fail

Return Single Type

One function should return only one type, eg. only string, hash, or floats. not float for one set of input and string for another set of input

If your function is returning different data types for different inputs then you might want to break it down into multiple functions or unify the returned data type.

Having multiple return types make it hard to test, and it can mean that we are not following single responsibility principle.

Multiple return type is considered a code smell, and should be avoided

Refactoring

It should not be left for later, refactor whenever you can, and when it feels the right thing to do.

Refactoring in small dosages keeps tech debt in check.

Closing Note

You want to ship faster but ship mindfully, and to ship faster you want to get your changes merged without lots of back and forth. You can do that by thinking and reasoning about how and why you are writing something before you start writing something.

Code less, engineer more by Liz Fong-Jones explains this very well, and It’s a must-read, this is something that I follow, and I try to not start coding till I understand the Why, What and How for the change

I suggest doing one round of code review yourself before the code review. I call this self-review, and I do it for all of my pull requests, docs, or anything that I share, and I always find things to improve, and I am sure you will also find things to improve too.

We should keep in mind that, these are guidelines, and they should not be taken to the extreme, our goal is to find a good balance, instead of bike-shading over these things guidelines, and we should focus on things that matter

Our goal is to solve problems and create value. our users don’t care if we are serving them using PHP, and jQuery or the hot new magical framework, as long as these things are not hindering user experience, your users don’t care.

CPU steal time and other caveats of virtualization

Thu, 11 Jun 2020 00:00:00 +0000

In this post, we will talk about downsides of virtualization in the context of cloud environments.

Cloud is great, we make an API call, and pay as we go, what’s not to like

Well, few things, which are obviously not mentioned in marketing materials, and are not commonly talked about on the web as much.

We will talk about CPU Steal Time and it’s partners in crime in virtualization

CPU Steal Time

Understand that your hardware is not real.

Wait, so they lied to me that I have 16 core machine?

Not really, to be correct you have 16 Virtual cores, the same physical machines have way more the 16 cores and it is shared with other VMs

Why do I care, I have the cores that I paid for?

Well, as I said they are virtual cores so other workload running on it can impact you.

How?

Allow me to introduce CPU steal time, the hypervisor can steal your CPU when it needs to do other important stuff.

Steal time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

– ibm.com

Further Reading:

The Noisy Neighbor Problem

Other misbehaving workloads running on the same physical hardware can impact things that hypervisors can’t set constraints on, like cache, network links, etc.

This is something to keep in mind when running on cloud environments

Further Reading:

Virtualization’s Impact on Performance Management

Security Implications

Side-channel attacks are a real threat when running in the cloud, research shows that it’s very much real.

Meltdown and Spectre showed us that speculative execution based attacks are real, and new research in this direction shows that this threat is very real.

Further Reading:

Hello from the Other Side: SSH over Robust Cache Covert Channels in the Cloud: Paper, Talk, and Media Coverage
Hardware vulnerabilities in cloud-native environments
Leveraging KVM Events to Detect Cache-Based Side Channel Attacks in a Virtualization Environment
A Placement Vulnerability Study in Multi-Tenant Public Clouds

Dedicated VMs

All cloud providers have a way to provision dedicated VMs(GCP calls then sole-tenant nodes) where physical machine runs only your workload.

They are expensive, but this is an option you have when you want extra security when running on cloud environments.

Closing Note

Cloud has more upsides than downsides but we should be aware of downsides and our options when we have to make a call where we can’t live with one of the downsides

Send your questions and corrections on twitter @electron0zero

Hopefully, it was helpful. Stay in and Stay Safe 👋

A Dash of Entertainment in Education

Fri, 22 May 2020 00:00:00 +0000

In this post, we will talk about how the educators are taking a page out of films production studies, and why it’s a good thing

I ran into Webflow tutorial and noticed that it is way better than most tutorials, so as promised here is my analysis.

We see that most tutorials are direct and pretty boring but they get the job done.

Few online educators are focused more on pure content¹, and the material, they don’t really pay attention to how it’s presented²

If we look at educators admired by Internet, David J Malon, and Andrew Ng comes to mind. On close observation we see that along with good material, they focus on keeping students engaged, try to keep it casual and entertaining by sharing relevant insights, and that changes the mood and turns the dreadful lecture into more engaging and enjoyable.

As a result, we see that students love these classes, and it can be seen in online communities, In ML community Andrew Ng memes are a thing 🙂

We see this new trend where educators are combining education and entertainment to engage users, but this is a tight rope because overdoing can feel forced and will lose the humor and then goes on the cringe side.

Webflow is using these techniques and looks like it’s working because it had me engaged. I watched multiple Webflow tutorials, and they are well designed, engaging, and hits the nice balance of education and entertainment

See this short tutorial on “Introducing scheduled publishing for Webflow CMS items” to better understand what I am talking about.

Anatomy of a Webflow tutorial

Set expectation of viewers by stating what the tutorial will teach, shows steps
Shows the parts/subparts, and reads them out loud to enforce it (add effect, don’t just read it out)
After each step, shows that the next step is happening by saying that step out loud and also showing it on screen
End with a punchline

This structure is good because it respects users, and provides good user experience³

It sets the expectation at the start, and it allows you can click away if you landed on the wrong tutorial.
Each step is clearly marked with visual and auditory indicators to ensure that users are aware of where they are
A dash of humor that acts like glue in those otherwise dry steps

Other Observations

Good writing is the key here, these tutorials are written like company sketch/ movies with focus on teaching and engagement
Music and sound design is another important thing⁴ that is done well
- No boring ukulele song, there are proper sound effects with good background music
Edits are done well, they incorporate visual gags, and have focus/zoom and other effects to keep it interesting
Tutorial opens up with the logo and sets the expectation, and then get to the point with few subtle gags to keep it interesting, and then end with a call to action of a punchline, and final screen is linked with a short call to action.

Closing Note:

Maybe, I over-analyzed the things but I seem to enjoy this new trend, and looks like others are liking it too, these are YouTube comments from one of Webflow video

In this thread, David Perell goes on more details, recommended reading 🙂

Update: Looks Netlify has entered the chat, see this and this tweet

That’s all, Stay In and Stay Safe 👋

footnotes:

they are still great and valuable, and I understand that not everyone has the budget for all the fancy things ↩
It depends on the nature of material and subject as well ↩
Videos have two dimensions of UX because they are experienced with eyes and ears, so if they are designed with both eyes and ears in mind, I consider it good UX ↩
Sound design can make and break a movie, and the same is true here ↩

Why do I use Dropbox Paper?

Thu, 14 May 2020 00:00:00 +0000

Twitter asked for details on why I use Dropbox Paper, I was writing the reply but ran out words so decided to turn it into a blog post 🙂

Why? Simple question but I am hoping for an insight here. –ProfADK

TL;DR:

It’s simple and has all the basic things I need, It has lots of small things that it got right, and it fits nicely with my workflow, and on top of that, it’s free 🤑

Long Answer 👇

In the past, I have tried using Notion, Google Docs, Typora, and all of them had something or another thing that I didn’t like, each one had few paper-cuts that annoyed me.

Tools are built for a use-case and user in mind, and those tools didn’t fit my use-case.

Reasons I like it:

Markdown support for writing, this is the biggest reason I like it because I am used to writing markdown
Can export Markdown, for me this makes publishing very easy on my Jekyll based blog
Have good support for rich media, GIFs, videos, and tweets have inline preview support
Slack style emojis, including custom emojis and I, prefer not using emoji picker (looking at you twitter 👀 )
Best writing experience for desktop, and mobile, I take lots of rough notes on my phone and then edit and refine it on desktop
Has small built-in tools like to-do’s(with the owner, due date and email reminders for missed to-do’s), timeline and tables
Decent support for code blocks
Can turn a post into a presentation with a button, saves time during internal talks where I turn my bullet points into a nice looking presentation
Best collaboration experience, I just share my drafts with a link of and people can review/comment on it on mobile
Folder-based document organization
Document template support (I don’t use them but it’s nice to have)
Search is very good and fast
Can link docs (just + and suggestions shows up) in other docs

Not So Good Things:

No advanced grammar and spell check, as a workaround, I do grammar fixes during edit stage using Grammarly
Document organization is basic but I can live with nested folders :)
It doesn’t work offline in the browser(on phone app it’s fine), when disconnected you can’t write/edit which is annoying
No themes :)

Workflow

Usual flow for a post goes like this:

Note down things, I call it brain dump (mostly bullet points)
Keep adding things, and It’s moved to drafts folder when I want to publish it someday
Expand it, put it into a structure
Do the first round of edits for grammar and spelling fixes
Send drafts(post links) to friends for review
Edit, and accommodate feedback
Download markdown file, move it to blog and add Jekyll front matter
git push and it’s live on the web 🎉

Closing Note

It’s free, syncs with my phone and has overall good experience, and fits nicely in my simple and minimal workflow.

In my books, Dropbox Paper is a well-designed product, and again, did I mention it’s free 🤑

That’s all, Stay in and Stay Safe 👋

Story of mysterious weekend restarts

Tue, 12 May 2020 00:00:00 +0000

In the early days of Clarisights(mid-2018), we had this problem where our Sidekiq VM used to restart on some weekends

We used to use Sidekiq to process and ingest user-uploaded CSV files into our data warehouse.

User flow goes like this

Users will upload big CSV files or we pull these daily from their servers(metabase)
We start a Sidekiq job to process to these in background
Sidekiq was configured to retry in case we had transient failures

It looks simple, right?

On one fine Monday, we got the issue from a customer that their data for Sunday was not processed. After some digging, we found that our Sidekiq VM restarted on Sunday, and it came back online automatically

We saw it restarted because it was out of memory, we attributed it to a bunch of retries(due to some bad CSVs) we saw around the same time and moved on

Next weekend it was fine, and then after a few more weeks, it happened again. we tried debugging but no luck, we added more instrumentation and moved on.

Fast forward to next week, while I was optimizing CSV processing code, I found a memory leak. CSV processing code used to leak memory, but it was a slow leak. Now all of sudden it all adds up on why it only happened one some weekends.

We used to deploy multiple times in a day, and during deploy, we restart Sidekiq to load new code.

Because of these multiple deploys, the leak was hidden and never showed up in monitoring. It was clear when we looked at restarts, memory growth, and deploys. , these restarts happened on weekends with zero deploys.

We ran an isolated test to verify it, and sure enough, it was the reason. So, that was my mysterious weekend restarts story, a slow memory leak, hiding behind deploys 🙂

For me, this was a perfect reminder of correlation can lead to causation but, correlation is not causation

That’s all folks, Stay In, Stay Safe 👋

Sending Kubernetes events to Sentry

Sun, 10 May 2020 00:00:00 +0000

In this post, we will talk about how to collect events from multiple Kubernetes clusters and ship it to sentry

At Clarisights, we run multiple Kubernetes clusters (GKE) for our production and staging workloads.

Events from Kubernetes objects get garbage collected to avoid putting extra load on etcd

events are stored in etcd with objects, and looks like it will stay that way for a while

When things blow up (like they do), we need those events to debug and fix the issues, so we wanted to put those events somewhere more permanent so we can come and look at it.

While searching for ways, we ran into sentry-kubernetes project, which collects and sends cluster events to sentry.

We already had self-hosted Sentry installation running, so this is just perfect, but upon looking closer we found that it sends events from a single cluster into a sentry project.

We wanted to send events from multiple clusters to a single project because we run multiple clusters and new clusters are spawned and moved around for various operational reasons.

Since code is open, we decided to add support for multiple clusters, we forked and sent a patch, yay open source 🙌

Now, let’s put it in production 🙌

Create a Sentry Project

Create a new sentry project, select Other under language/framework
Copy DSN, we will need it later

Setup Access

To collect these events we need permissions to watch these events, we need to setup access so our collector and watch, collect and send these events to sentry

For access we will create a ServiceAccount, a ClusterRole with permissions to watch events, and bind that role with ServiceAccount using a ClusterRoleBinding.

If you are new to Kubernetes RBAC, see my post on it to learn more, and see this gist for yaml file

We will use this service account in our deployment

Deployment

Build docker image
- docker build -t sentry-kubernetes:latest https://github.com/getsentry/sentry-kubernetes.git
Create and apply Deployment with sentry-kubernetes image and service account we created, and if all goes well we should see events in sentry project.

Closing Note

Cluster name, namespace and other Kubernetes object attributes will show up as labels, and you should be able to filter events based on these.

See this gist for more scripts and yaml files.

I hope it was helpful :)

Stay in and Stay Safe 👋