

![Coffee Filter - Selective Filtering](/images/selective-redeployments-cloud-functions/hero.png)

[Firebase Functions](https://firebase.google.com/docs/functions) are a convenient wrapper around [Google Cloud Functions](https://cloud.google.com/functions).

My team at [upcover](https://upcover.com) started using them to prototype and deploy APIs on Google Cloud.
They provide an easy local development experience with the [Firebase Emulator Suite](https://firebase.google.com/docs/emulator-suite), and a convenient deployment workflow with the `firebase` CLI.

## Issues

As our product grew in complexity, the number of Cloud Functions we maintained increased significantly. Once we hit around 60 Cloud Functions, we experienced two major issues:

1. The `firebase` CLI tries to deploy all Cloud Functions in your project in parallel. Given enough Cloud Functions, it leads to rate limit errors with the Google Cloud API.
2. The CLI deploys every Cloud Function every time, regardless of whether a function actually needs redeployment.

## Solving Rate Limit Errors

Google Cloud has [well defined quotas](https://firebase.google.com/docs/functions/quotas) for API calls. The `firebase deploy` command frequently exceeded this quota,
because it tried to deploy every Cloud Function in parallel.

To solve the API rate limit problem, we started deploying our Cloud Functions in batches of 30. I created a simple Node.js script that did the following:

1. Discover all Cloud Functions exported from the repository.
2. Partition discovered functions into batches of 30 (I settled on the number 30 arbitrarily).
3. Run the `firebase deploy --only functions:[<function-names>]` for the functions in each batch.

Here's a simplified implementation of the script:

```ts
import { chunk } from "lodash";
import { execSync } from "child_process";

const functionsIndex = require("./libs/src/index");
const exportedFunctions = Object.keys(functionsIndex);

const batches = chunk(exportedFunctions, 30);

for (const batch of batches) {
  const args = batch
    .map((functionName) => `function:${functionName}`)
    .join(",");

  const command = `firebase deploy --only ${args}`;
  execSync(command, { stdio: "inherit" });
}
```

The `functionsIndex` module is a compiled JavaScript file. It's produced from the `src/index.ts` TypeScript file containing all exported Cloud Functions.
Here's a sample of what it looks like:

```ts
// src/index.ts

export { auth } from "./services/auth/function";
export { quotes } from "./services/quotes/function";
export { documents } from "./services/documents/function";
```

**This script solved the rate limit errors, but led to longer CI builds**.

It ran `firebase deploy` multiple times: once for every batch of Cloud Functions. While this led to longer builds, it was needed because the deployments outright failed otherwise.

### Adopting Selective Redeployments

Every Cloud Function requires time to deploy. The more Cloud Functons you have, the longer your deployment time.
When our CI build hit a duration of 30 mins, we began investigating possible solutions.

Our CTO, Sajjad Naveed, pointed out that we could reduce the time by deploying only the Cloud Functions that changed.
To detect changed Cloud Functions, he suggested a way to calculate every function's identity.

**If the identity of a function does not change, it does not need redeployment.**

### Cloud Function Identity

Sajjad suggested bundling a Cloud Function, and then calculating a hash of the bundle to produce the its identity.

I created a script that performed the following steps:

1. Discover every Cloud Function in the repository.
2. Bundle the code of every Cloud Function using a bundler, with minification and tree shaking.
3. Calculate the hash of the bundled code.
4. Compare the newly calculated hash with an existing hash value stored in a local file. If unchanged, append the function to the list of functions that need redeployment. Finally, update the local file containing hashes with the new hash of this function.
5. Selectively deploy only the functions that need redeployment.

### Why Does This Work?

Consider the following Cloud Function:

```ts
import * as functions from "firebase-functions";
import { FooService } from "./fooService";

export const FooFunction = functions.https.onRequest((req, res) => {
  const message = FooService.getMessage();
  res.json({ message });
});
```

The identity of `FooFunction` can change due to three reasons only:

1. If we update the `firebase-functions` package
2. If we update the implementation of `FooService`
3. If `FooFunction` itself changes (if we change the response to `res.status(201).json({ message })`)

**Simply put, the identity of a Cloud Function changes only its code, or its dependencies change**

Bundling therefore works great for this purpose: the produced bundle for a Cloud Function would change when any/all of the above happen.
A different bundle would produce a different hash, and therefore, a new identity.

## Results

Selective redeployments dramatically reduced our **average** deployment time.

If a pull request affected only a single cloud function, merging that pull request would trigger redployment of that function only.
However, if a pull request updated a dependency used by many cloud functions, all of them would need redeployment.

Here are some screenshots to demonstrate:

![Short Selective Redeployment](/images/selective-redeployments-cloud-functions/short-selective-redployment.png)

<p className="text-center text-[0.8rem] text-gray-500 dark:text-gray-300">
  <i>Deployment Affecting a Single Cloud Function</i>
</p>

![Long Selective Redeployment](/images/selective-redeployments-cloud-functions/long-selective-redeployment.png)

<p className="text-center text-[0.8rem] italic text-gray-500 dark:text-gray-300">
  <i>Deployment Affecting Multiple Cloud Functions</i>
</p>

I used [ESBuild](https://esbuild.github.io) as the bundler (mainly due to its speed), but the process should work with any bundler.

## functions-differ

I decided to build this functionality into a reusable open source package.

Meet [`functions-differ`](https://npmjs.com/package/functions-differ). A CLI tool written in Node.js that calculates and maintains function identities. Use it to selectively redeploy your Cloud Functions.

It's an experimental tool that I maintain in my free time, so there are some rough edges. Feel free to try it and share feedback.

I plan to write a post on how to integrate `functions-differ` with your CI workflow in the future. Watch this space for more.

---

_Thanks to [Bhanu](https://github.com/hiiambhanu) and Sajjad for contributing to this tool._

_Questions, comments, or feedback? Reach out to me on Twitter: [@haroldadmin](https://twitter.com/haroldadmin)_


Selective redeployment of Cloud Functions for faster CI builds

Optimizing Deployments for Cloud Functions



![Hero Image](/images/cloudflare-jitpack-shields/hero.png)

I have published a few libraries on [Jitpack](https://jitpack.io).

I like Jitpack for its simplicity. Besides artifact hosting, it offers [release badges](https://jitpack.io/docs/#badges), and a free, public [API](https://docs.jitpack.io/api) to query statistics about published libraries. One such stat is the number of weekly and monthly [downloads](https://docs.jitpack.io/api/#download-statistics).

**I wanted to display the number of downloads as a badge on one of my library's README file**, but Jitpack does not offer badges for download statistics.

Given the lack of official support, I started looking for alternatives and discovered [Shields.io](https://shields.io/), a platform that offers many different types of badges for free. Unfortunately, it does not have badges for Jitpack download stats either.
It does have the ability to build badges from custom API [endpoints](https://shields.io/endpoint), as long as the API responds with a fixed predefined schema.

---

**Shields.io Schema**

| Property        | Description                                                                                                                                                                                                                                                                                                                                 |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schemaVersion` | _Required_. Always the number 1.                                                                                                                                                                                                                                                                                                            |
| `label`         | _Required_. The left text, or the empty string to omit the left side of the badge. This can be overridden by the query string.                                                                                                                                                                                                              |
| `message`       | _Required_. Can't be empty. The right text.                                                                                                                                                                                                                                                                                                 |
| `color`         | _Default: lightgrey_. The right color. Supports the eight named colors above, as well as hex, rgb, rgba, hsl, hsla and css named colors. This can be overridden by the query string.                                                                                                                                                        |
| `cacheSeconds`  | _Default: 300_, min 300. Set the HTTP cache lifetime in seconds, which should be respected by the Shields' CDN and downstream users. Values below 300 will be ignored. This lets you tune performance and traffic vs. responsiveness. The value you specify can be overridden by the user via the query string, but only to a longer value. |

---

Here's what the response of Jitpack's download stats API looks like:

```json
{
  "week": 1707,
  "month": 22214
}
```

To create my badge, I needed to build a service that acts as an adapter between Jitpack and Shield.io.

![Flow](/images/cloudflare-jitpack-shields/flow.svg)

A serverless function felt like the natural fit for this task. I could use one of a number of available options:

- HTTPS Google Cloud Function
- Containerised Google Cloud Run service
- AWS Lambda

I decided I wanted to learn something new, so I went with a [Cloudflare Worker](https://workers.cloudflare.com/) instead.

Cloudflare offers a generous free tier, but I was surprised to learn about the 50ms CPU time limit on a worker.
Even if Jitpack's API was extremely fast, it would surely take more than 50ms to fetch download stats for an artifact.

After reading some docs, I realized the time limit is for **active CPU usage only**. Most of my worker's time would be spent awaiting a `fetch()` API call, and thankfully that won't count as active CPU time.

Cloudflare's quickstart guide was enough to teach me everything I needed to build the worker. Ignoring all the setup with the `wrangler` CLI, here's what the worker looks like:

```javascript
const baseUrl = "https://jitpack.io/api";

addEventListener("fetch", (event) => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const url = new URL(request.url);

  const group = url.searchParams.get("group");
  const artifact = url.searchParams.get("artifact");
  const frequency = url.searchParams.get("frequency");

  let res;
  switch (frequency) {
    case "monthly":
      res = await getMonthlyDownloads(group, artifact);
      break;
    case "weekly":
      res = await getWeeklyDownloads(group, artifact);
      break;
    default:
      return new Response(null, { status: 400 });
  }

  return new Response(JSON.stringify(res), {
    headers: { "content-type": "application/json" },
  });
}

async function getMonthlyDownloads(groupId, artifactId) {
  const downloadsUrl = `${baseUrl}/downloads/${groupId}/${artifactId}`;
  const response = await fetch(downloadsUrl);
  const { month: monthlyDownloads } = await response.json();

  const res = {
    schemaVersion: 1,
    label: "Downloads",
    message: `${monthlyDownloads}/month`,
    color: "green",
  };

  return res;
}

async function getWeeklyDownloads(groupId, artifactId) {
  const downloadsUrl = `${baseUrl}/downloads/${groupId}/${artifactId}`;
  const response = await fetch(downloadsUrl);
  const { week: weeklyDownloads } = await response.json();

  const res = {
    schemaVersion: 1,
    label: "Downloads",
    message: `${weeklyDownloads}/week`,
    color: "green",
  };

  return res;
}
```

I then deployed it with `wrangler publish`, and then created a custom shield on Shields.io.

The result: ![Result](/images/cloudflare-jitpack-shields/result.png)

I'm pretty happy with how it turned out. This was my first experience with Cloudflare Workers, and it left a positive impression. I might start using it more in the near future.

---

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_


Creating a Shields.io badge for Jitpack downloads using Cloudflare Workers

Experiments with Cloudflare Workers



![Hero Image](/images/joi-data-transformation/hero.png)

The Node.js community has access to many data validation libraries like [Ajv](https://www.npmjs.com/package/ajv), [Yup](https://www.npmjs.com/package/yup), [class-validator](https://www.npmjs.com/package/class-validator), [validate.js](https://www.npmjs.com/package/validate.js), and more. One library that stands out to me is [Joi](https://www.npmjs.com/package/joi).

Joi is popular, battle tested, and very powerful, but it's often mistaken to be "just" a data validation library even though it can also transform data. This ability is really useful when building REST APIs, as I'll demonstrate using a combination of Nest.js and Joi.

Consider the following Nest.js `Pipe` that validates an incoming request body using a Joi schema:

```ts
import { Schema } from "joi";
import { PipeTransform, BadRequestException } from "@nestjs/common";

class SchemaValidator implements PipeTransform {
  constructor(readonly schema: Schema) {}

  transform(incoming: unknown): unknown {
    const { error } = this.schema.validate(incoming);
    if (error) {
      throw new BadRequestException(error.message);
    }

    return incoming;
  }
}

// Convenient way to apply the SchemaValidator
function useSchema(schema: Schema): SchemaValidator {
  return new SchemaValidator(schema);
}
```

It runs the incoming value through the given schema to validate it. It then returns a "400 Bad Request" response if the validation is unsuccessful, or returns the value as-is otherwise. You can apply it as any other pipe:

```ts
@Controller()
class UserController {
  @Get("/:email/picture")
  getProfilePicture(
    @Param("email", useSchema(Joi.string().email().required()))
    email: string
  ): string {
    // ...
  }
}
```

A common practice when working with emails is to lowercase them before usage. The simplest way to do this would be to call `email.toLowerCase()` in the controller method, but you could also leverage Joi's capabilities for this.

First, apply the `lowercase` transformation to the schema:

```ts
const emailSchema = Joi.string().email().lowercase().required()

@Get("/:email/picture")
getProfilePicture(
  @Param("email", withSchema(emailSchema)) email: string
) {
  // ...
}
```

Then adjust the `transform` method of `SchemaValidator` pipe to return Joi's validated value instead of the original value:

```ts
transform(incoming: unknown): unknown {
  const { error, value } = this.schema.validate(incoming);
	if (error) {
		throw new BadRequestException(error.message);
	}

  return value;
}
```

This ensures that the value returned from this pipe will be lowercase as long as the `convert` option is enabled (it's enabled by default). You can use any of the other available transformations like `default()`, `sort()`, `binary()` when needed, and you could also apply custom transformations using your own functions.

It's a best practice to normalize emails before usage. `foo@gmail.com` and `foo+bar@gmail.com` are the same email addresses from Gmail's perspective. Similarly, `foobar@gmail.com` and `foo.bar@gmail.com` point to the same email. It's therefore important to normalize email variants before use through a package like [normalize-email](https://www.npmjs.com/package/normalize-email).

First, write a custom validator for your email schema that uses the `normalize-email` package:

```ts
import { CustomHelpers, ErrorReport } from "joi";
import normalize from "normalize-email";

function normalizeEmail(
  value: string | undefined,
  helpers: CustomHelpers
): string | ErrorReport {
  try {
    const normalized = normalize(value);
    return normalized;
  } catch (error) {
    return helpers.message({ custom: "Failed to normalize email" });
  }
}
```

Then modify your controller to use it:

```ts
const emailSchema = Joi.string()
  .email()
  .lowercase()
  .required()
  .custom(normalizeEmail)

@Get("/:email/picture")
getProfilePicture(
  @Param("email", withSchema(emailSchema)) email: string
) {
  // ...
}
```

And now you will always received normalized emails!

---

If you liked this post and want more content on Joi, let me know with a tweet [@haroldadmin](https://twitter.com/haroldadmin)!


Joi for Data Transformations



![Hero](/images/asynclocalstorage-logs-nestjs/hero.png)

Logging is invaluable in backend services. Structured logging makes logs more useful by enabling you to categorize, analyze and search logs effectively. Sadly though, setting up structured logging correctly is difficult, especially in dependency injection-oriented frameworks such as [Nest.js](https://nestjs.com/).

This blog post is code-snippet heavy. It's the most succinct way to convey the problem I'm trying to solve, so prepare to read a lot of code.

Consider the following `AuthService` that validates a user's credentials and logs information using a hypothetical `StructuredLogger`:

```typescript
@Injectable()
class AuthService {
  constructor(
    private readonly logger: StructuredLogger,
    private readonly auth0: Auth0Service
  ) {}

  validate = async (creds: Credentials): Promise<string | Error> => {
    this.logger.info({ ...creds }, "Validating credentials");

    const result = await checkCredentials(creds);
    if (!result.success) {
      this.logger.error({ error: result.error }, "Invalid credentials");
      return result.error;
    }

    this.logger.info("Credentials valid");
    return result.accessToken;
  };
}
```

The `validate` method produces the following log statment whenever it runs:

```
[2021-07-11 06:56:49.574 +0000] INFO: Validating credentials
    username: "johndoe"
    scopes: [
      "offline profile email"
    ]
    redirectUrl: "https://example.com/auth/redirect"
```

It is useful because it contains the arguments passed to the `validate` method. A clever log parser can then process this line to separate the log message (_"Validating credentials"_) and the log metadata (_username, scopes, redirectUrl_) to produce clean logs.

## Context of Execution

While the above log statement is alright, it can be better. A key piece of information missing from it is its context of execution, such as the name of the service and the method that produced it, as well as the ID of incoming request that triggered the service.

### Service Name

It's easy to add the name of the service to the logger. Most logging packages support extending a parent logger with some default metadata that is appended to every log line. Therefore we can inject a common base logger, and extend it inside our service's constructor with the service name.

```typescript
@Injectable()
class AuthService {
  private readonly logger: StructuredLogger;

  constructor(
    baseLogger: StructuredLogger,
    private readonly auth0: Auth0Service
  ) {
    this.logger = baseLogger.child({ serviceName: AuthService.name });
  }

  validate = async (creds: Credentials) => {
    this.logger.info({ ...creds }, "Validating credentials");
    // ...
  };
}
```

Now the same log statement includes the service name as well:

```
[2021-07-11 06:56:49.574 +0000] INFO: Validating credentials
    serviceName: "AuthService"
    username: "johndoe"
    scopes: [
      "offline profile email"
    ]
    redirectUrl: "https://example.com/auth/redirect"
```

### Method Name

You can extend the service's logger inside every method to append the `methodName` property to the logger's metadata:

```typescript
validate = async (creds: Credentials) => {
  const methodLogger = this.logger.child({ methodName: "validate" });
  this.logger.info({ ...creds }, "Validating credentials");
  // ...
};
```

Now the same log line contains the `methodName` attribute as well:

```
[2021-07-11 06:56:49.574 +0000] INFO: Validating credentials
    serviceName: "AuthService"
    methodName: "validate"
    username: "johndoe"
    scopes: [
      "offline profile email"
    ]
    redirectUrl: "https://example.com/auth/redirect"
```

### Request ID

The ID of a request is typically contained inside the `x-request-id` header in an incoming request. Therefore, the simplest way of adding it to the logs is to pass the incoming request to the `validate` method, and let it extend its parent logger with a `requestId`.

```typescript
validate = async (creds: Credentials, req: Request) => {
  const requestId = req.headers["x-request-id"];
  const correlationId = req.headers["x-correlation-id"];
  const methodLogger = this.logger.child({
    methodName: "validate",
    reqId: requestId,
    cid: correlationId,
  });
  // ...
};
```

Now the logs include the request ID too:

```
[2021-07-11 06:56:49.574 +0000] INFO: Validating credentials
    serviceName: "AuthService"
    methodName: "validate"
    reqId: "4d426f0a-cf13-47de-a824-f98d98ff7e83"
    cid: "fa38e3d4-6fc0-4bd6-b5cd-cb9f43f994f8"
    username: "johndoe"
    scopes: [
      "offline profile email"
    ]
    redirectUrl: "https://example.com/auth/redirect"
```

It works, _but not really_.

1. It pollutes the signature of the `validate` method, which should only be concerned with the credentials argument.
2. It makes `AuthService` harder to test, as now every test must stub or mock the `req` parameter when testing the `validate` method.
3. It's terribly inconvenient. Extending a logger at every step of execution is a fragile process, and someone on your team can easily forget to do so.

## Injecting a Customized Logger

An alternative solution to this problem is to inject a customized logger using Nest.js that is scoped to a request:

```typescript
export const LoggerProvider: Provider<StructuredLogger> = {
  provide: StructuredLogger,
  scope: Scope.REQUEST,
  useFactory: () => new StructuredLogger(),
};
```

You might be able to do some [ExecutionContext](https://docs.nestjs.com/fundamentals/execution-context) magic and inject a request's ID into this logger, but I'm not sure how. Either way, this solution has the following problems:

1. You're forced to inject a logger instance, even though loggers should be created and destroyed freely.
2. This provider is limited to a _Request Scope_, which means it is recreated on every request. It implies that any service or controller that depends on this provider will **also be request scoped**, regardless of any explicitly specified scope, thus causing needless object allocations on every request.

## Enter: `AsyncLocalStorage`

[`AsyncLocalStorage`](https://nodejs.org/api/async_context.html#async_context_class_asynclocalstorage) is a feature introduced in Node v14 as a part of the `async_hooks` package. It allows you to access an implicitly passed object anywhere in an asynchronous call chain. Its canonical use case is to keep a track of a request ID throughout the request's lifetime.

> While the `async_hooks` API is unstable, the `AsyncLocalStorage` class is considered stable in Node.js. It has been backported to Node.js v12 too.

### Middleware

You can create a functional middleware to create a request context for every incoming request with `AsyncLocalStorage`.

- First, define a global store for storing contexts of every incoming request:

```typescript
interface RequestContext {
  [key: string]: string;
}

const globalStore = new AsyncLocalStorage<RequestContext>();
```

- Then create a couple of convenience methods that enable wrapping a request with a context and accessing that context within the request:

```typescript
// Allows easy access to a request's context
const ctx = (): RequestContext => {
  const context = globalStore.getStore();
  if (!context) {
    throw new NoContextError("No context attached to the current req");
  }
  return context;
};

// Allows wrapping a request in a context
const runWithCtx = (
  fx: (ctx: RequestContext) => Promise<unknown>,
  context: RequestContext = {}
) => {
  globalStore.run(context, () => {
    return fx(ctx());
  });
};
```

- Then create a Nest Functional Middleware that wraps every request in a context:

```typescript
const withContext = (_req: Request, _res: Response, next: NextFunction) => {
  runWithCtx(() => next(), {});
};
```

- Then create a middleware that stores a request's `x-request-id` and `x-correlation-id` headers in the request context:

```typescript
const withRequestId = (req: Request, _res: Response, next: NextFunction) => {
  const context = ctx();
  context.correlationID = req.headers["x-correlation-id"];
  context.requestID = req.headers["x-request-id"];
  return next();
};
```

- Finally, apply the `withContext` and `withRequestId` middleware as early as possible in the Nest application's middleware chain. Ensure that `withContext` executes before `withRequestId`:

```typescript
async function main() {
  const app = await NestFactory.create(AppModule);
  app.useLogger(withContext, withRequestId);
  await app.listen(3000);
}
```

With this setup complete, you can now access the request context implicitly inside any service or controller. I recommend creating a logger factory method to automatically initialize a logger with such details:

```typescript
const createLogger = (
  serviceName: string,
  methodName: string
): StructuredLogger => {
  const context = ctx();
  const { requestId, correlationId } = context;
  const baseLogger = new StructuredLogger();
  const customizedLogger = baseLogger.child({
    serviceName,
    methodName,
    requestId,
    correlationId,
  });
  return customizedLogger;
};
```

### Usage

The `createLogger` method greatly simplifies the logging setup in `AuthService`:

```typescript
@Injectable()
class AuthService {
  constructor(private readonly auth0: Auth0Service) {}

  validate = async (creds: Credentials): Promise<string | Error> => {
    const logger = createLogger(AuthService.name, "validate");
    logger.info({ ...creds }, "Validating credentials");

    const result = await checkCredentials(creds);
    if (!result.success) {
      logger.error({ error: result.error }, "Invalid credentials");
      return result.error;
    }

    logger.info("Credentials valid");
    return result.accessToken;
  };
}
```

1. You no longer need a `logger` parameter in the constructor.
2. The `createLogger` method simplifies extending a base logger with metadata.
3. Since the logger is not a class property anymore, you don't need to access it with `this.logger`.

## Performance Caveat

I like this log setup, but as with everything else in programming, it's a compromise. `AsyncLocalStorage` has a non-negotiable performance hit. It's quite small, but measurable. Make sure to benchmark your setup to check if the convenience is worth the performance penalty.

Here are the results from a rudimentary benchmark for an endpoint that returns _"Hello world!"_ to every request, using the [Apache Benchmarking tool](https://httpd.apache.org/docs/2.4/programs/ab.html). I ran the test on an Apple M1 MacBook Air running Node v16.1.0.

![Request Throughput](/images/asynclocalstorage-logs-nestjs/req-per-sec.svg)
![Request Latency](/images/asynclocalstorage-logs-nestjs/mean-time-per-req.svg)

- Without `AsyncLocalStorage` middlware:

```shell
$ ab -n 1000 -c 100 localhost:3000/greet

Server Software:
Server Hostname:        localhost
Server Port:            3000

Document Path:          /greet
Document Length:        13 bytes

Concurrency Level:      100
Time taken for tests:   0.201 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      212000 bytes
HTML transferred:       13000 bytes
Requests per second:    4965.98 [#/sec] (mean)
Time per request:       20.137 [ms] (mean)
Time per request:       0.201 [ms] (mean, across all concurrent requests)
Transfer rate:          1028.11 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.0      0       3
Processing:     5   19   5.8     18      33
Waiting:        2   11   4.5     11      29
Total:          5   20   6.4     18      35
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     21
  75%     22
  80%     25
  90%     29
  95%     35
  98%     35
  99%     35
 100%     35 (longest request)
```

- With `AsyncLocalStorage` middlware:

```shell
$ ab -n 1000 -c 100 localhost:3000/greet

Server Software:
Server Hostname:        localhost
Server Port:            3000

Document Path:          /greet
Document Length:        13 bytes

Concurrency Level:      100
Time taken for tests:   0.219 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      382000 bytes
HTML transferred:       13000 bytes
Requests per second:    4565.08 [#/sec] (mean)
Time per request:       21.905 [ms] (mean)
Time per request:       0.219 [ms] (mean, across all concurrent requests)
Transfer rate:          1702.99 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.1      2       4
Processing:     4   19   8.4     17      41
Waiting:        3   11   6.2     10      38
Total:          5   20   8.9     18      43

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     21
  75%     21
  80%     26
  90%     41
  95%     42
  98%     42
  99%     42
 100%     43 (longest request)
```

4965.98 vs 4565.08 requests per second. I think I can take a loss of 400 requests per second for the added convenience, but you should decide what works for you based on your usecase.


Using request context for more detailed logs

AsyncLocalStorage and Structured Logging in Nest.js



![Hero](/images/circular-refs-kotlin/hero.png)

Circular object references can be dangerous. They break code at runtime, and are difficult to guard against at compile time.

Consider the following example:

```kotlin
data class Driver(val name: String, val teamId: String)

data class Team(val id: String, val drivers: List<Driver>)

object Drivers {
  val LewisHamilton: Driver = Driver("Sir Lewis Hamilton", Teams.Mercedes.id)
}

object Teams {
  val Mercedes: Team = Team("merc", listOf(Drivers.LewisHamilton))
}

fun main() {
  println(Drivers.LewisHamilton)
  println(Teams.Mercedes)
}
```

Although innocuous at first glance, **this code breaks Kotlin's non-nullable types**.

If you run this example on [Kotlin Playground](https://pl.kotl.in/Pri2qj09N), it produces the following output:

```text
Driver(name=Sir Lewis Hamilton, teamId=merc)
Team(id=merc, drivers=[null])
```

A `null` value snuck into a non-nullable type.

## How?

Kotlin `object`s are lazily initialized[^1].

`LewisHamilton` gets initialized upon its first-access in `main`. Its constructor references another lazy variable `Mercedes`, whose constructor has a circular reference to `LewisHamilton`.

Therefore, constructor of `Mercedes` references a yet-uninitialized `LewisHamilton` variable, and leads to a null value being stored in a non-nullable variable.

While this is bug is easy to fix, the problem is that this code fails at runtime, thus **silently breaking Kotlin's non-null types**.

Consider starring this issue to get more attention to this problem: [KT-44633: Circular object references silently break Kotlin's non-nullable types ](https://youtrack.jetbrains.com/issue/KT-44634)

## More Circular Reference Madness

Let's look at a few more examples to illustrate problems with circular references.

```kotlin
data class Foo(var bar: Bar? = null)

data class Bar(var foo: Foo? = null)

fun main() {
  val foo = Foo()
  val bar = Bar()
  foo.bar = bar
  bar.foo = foo

  println(foo)
}
```

This code produces a `StackoverflowError`. You can try it [here](https://pl.kotl.in/wJdoP8NQ6).

`println` calls `toString()` on `foo`. Since `Foo` is a data class, its `toString()` implementation calls the same method on its member properties too. This leads to an ever growing call-stack.

```
⤷ `Foo.toString()`
  ⤷ `Bar.toString()`
    ⤷ `Foo.toString()`
      ⤷ `Bar.toString()`
        ⤷ `Foo.toString()`
          ⤷ ...
```

A simpler, more condensed version of the same problem can be illustrated as follows:

```kotlin
data class CircularRef(var ref: CircularRef? = null)

fun main() {
    val circularRef = CircularRef()
    circularRef.ref = circularRef
    println(circularRef)
}
```

This code again produces a `StackoverflowError`, as `CircularRef.toString()` continues recursing forever. Try it [here](https://pl.kotl.in/cDfjeEkd7).

## Not just Kotlin

The problem of Circular References plagues other languages too. Here's an example of the same code in Go.

```go
type Foo struct {
	BarRef *Bar
}

func (f *Foo) String() string {
	return fmt.Sprintf("Foo { BarRef: %v }", f.BarRef)
}

func (b *Bar) String() string {
	return fmt.Sprintf("Bar { FooRef: %v }", b.FooRef)
}

type Bar struct {
	FooRef *Foo
}

func main() {
	foo := Foo{}
	bar := Bar{}
	foo.BarRef = &bar
	bar.FooRef = &foo
	fmt.Println(foo)
	fmt.Println(bar)
	fmt.Println("Finished")
}
```

Note circular references in the `String()` method on both interfaces. Run this [here](https://play.golang.com/p/4VlvFjfFkNm).

This code never prints _Finished_. While the execution timeout on _play.golang.org_ prevents this code from running for long, the same code on my local machine prints: `fatal error: stack overflow`.

## Conclusion

Be on the look out for circular references in your Kotlin code. You will receive no compile time errors or warnings about them.

If you would like to change that, consider starring this issue: [KT-44634: Circular object references silently break Kotlin's non-nullable types](https://youtrack.jetbrains.com/issue/KT-44634).

---

[^1]: https://kotlinlang.org/docs/reference/object-declarations.html#object-declarations

_Thanks to Subhrajyoti Sen for reviewing this post!_

---

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_


Exploration of how circular references break null-safety in Kotlin

Breaking Kotlin's Null-Safety with Circular References



![Hero](/images/escape-fts-queries/hero.png)

[Full Text Search](https://www.sqlite.org/fts3.html) (_FTS_) is a powerful feature of SQLite, but its intricacies sometimes make it a footgun.

> **Footgun** (_noun_) <sup>[_source_](https://en.wiktionary.org/wiki/footgun)</sup>
>
> _Any feature whose addition to a product results in the user shooting themselves in the foot._

Say you have a database of SpaceX launches, and you want to use FTS on it to power the search experience in an application. You create the FTS table, you build a nice UI around it, and it works perfectly. _Super easy!_

Then you notice whenever you search for anything that starts with a dash/minus-sign (`-`), your application crashes with an error like this:

```
android.database.sqlite.SQLiteException: malformed MATCH expression: [*-6*] (code 1 SQLITE_ERROR)
        at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
        at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:942)
        at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:838)
        at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
        at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:153)
        at android.database.sqlite.SQLiteCursor.getCount(SQLiteCursor.java:140)
```

In this case, the search query was _'-6'_ to search for the [Starlink-6](https://www.rocketlaunch.live/launch/starlink-6) launch.

This happened to me recently. Upon investigation, I realized that I had shot myself in the foot with FTS because of its [Set Operations](https://www.sqlite.org/fts3.html#set_operations_using_the_standard_query_syntax) feature.

## FTS Set Operations

Set Operations allow users to combine the results of multiple FTS queries in different ways. Essentially, they allow you to use logical `AND`, `OR` and `NOT` operators in search queries.

Set Operations treat the unary minus symbol `-` as the logical `NOT` operator. When used correctly, it can be used to retrieve the complement result set (i.e., all results _minus_ the results for the given query). Additionally, any queries prefixed with `-` are forbidden, and SQLite throws an error when it encounters one (which explains the earlier crash).

Let's say you want to search for all Starlink launches, except the ones which contain _'8'_ in their name. You can accomplish this by simply prefixing the unwanted term with a `-`. The search query then becomes _'Starlink -8'_.

_...But wait!_ This is also the exact query you would to search for the [Starlink-8](https://www.rocketlaunch.live/launch/starlink-8) launch! But FTS interprets it completely differently!

To fix this problem, you must escape such characters in the search query.

## Escaping search queries

The simplest way to escape special characters in the user's search queries is to enclose them in double quotes. This ensures that SQLite treats the search query literally.

```kotlin
fun search(query: String) {
  db.search("\"$query\"") // Starlink-8 -> "Starlink-8"
}
```

Double quotes ensure that SQLite treats the search query literally.

Unfortunately, it's not enough. This code breaks if the input query contains double quotes too.

## Escaping search queries even more

SQLite's documentation clearly defines [verbatim strings for FTS5](https://sqlite.org/fts5.html#fts5_strings), and through experimentation I can confirm that the definition applies to FTS3 and FTS4 too.

According to the docs, any `"` in the query should be replaced with `""`, and then the whole thing must be wrapped in double quotes [^1]:

```kotlin
fun search(query: String) {
  val escapedQuery = query.replace(Regex.fromLiteral("\""), "\"\"")
  db.search("\"$escapedQuery\"") // Starlink"-8 -> "Starlink""-8"
}
```

This solution seems to work for most cases. I shall update this post if I discover any queries that break this code.

Search query manipulation can still be performed outside the quotes. For example, to match the query when it's in the middle of a word, surround it asterisks:

```kotlin
fun search(query: String) {
  val escapedQuery = query.replace(Regex.fromLiteral("\""), "\"\"")
  db.search("*\"$escapedQuery\"*") // Starlink"-8 -> *"Starlink""-8"*
}
```

---

Given the complexity of sanitizing user input, it makes more sense to expose Set Operations related functionality of FTS through controllable UI elements (such as buttons and toggles), rather than an uncontrolled text field.

In conclusion, always sanitize user input and escape your FTS search queries.

If you want learn more about the Full Text Search feature in Room, check out the tutorial I did for raywenderlich.com: [Introduction to Full Text Search in Room](https://www.raywenderlich.com/14292824-introduction-to-full-text-search-in-room).

---

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_

[^1]:
    Here's a handy extension function to perform this sanitization on a String:
    `fun String.sanitize(): String = "\"${replace(Regex.fromLiteral("\""), "\"\"")\""`
    You can [try it here](https://pl.kotl.in/unazGaBQm).

---


Performing proper input sanitization on FTS search queries in SQLite

Escape your Full Text Search queries



![Hero](/images/finding-right-path/hero.png)

On a recent project, I got the opportunity to create a desktop application using [Electron](https://electronjs.org), [React](https://reactjs.org) and [Go](https://golang.org). Electron served as a tool to bundle the React app and the Go server in a neat little installable package. It also took care of starting the Go server as a subprocess when the app launched.

The included Go server performed a few dependency checks at each startup by trying to find some particular executables in the `PATH`.

```go
func checkDependency(cmd string) bool {
  _, err := exec.LookPath(cmd)
  return err == nil
}
```

This worked great during the development period, but once I built an installable package and installed the app, these dependency checks failed every time I launched it.

_Curiously, the checks only failed only on MacOS and Linux, not on Windows._

_Even more curiously, launching the installed app through a terminal seemingly fixed this problem._

The reason for this behaviour, as I found out after several hours of painful debugging, was unexpected inheritance of the `PATH` environment variable.

## Inheritance of Environment Variables

**A newly spawned child process inherits its parent's environment variables.**

To demonstrate this, let's build a couple of Go programs: `envReader` and `envParent`.

`envReader` reads a special environment variable _"SpecialEnv"_ and prints its value, or _"\<Not Found\>"_ if it is not set.

```go
// envReader
func main() {
  val := os.Getenv("SpecialEnv")
  if val == "" {
    val = "<Not Found>"
  }
  fmt.Printf("%d: SpecialEnv: %s", os.Getpid(), val)
}
```

`envParent`, runs `envReader` as a subprocess twice, with the "SpecialEnv" set only in the second invocation.

```go
// envParent
func main() {
  fmt.Println("Before setting special env:")
  runChild()

  fmt.Println("Setting SpecialEnv in parent")
  os.Setenv("SpecialEnv", "SpecialValue")

  fmt.Println("After setting special env:")
  runChild()
}

func runChild() {
  child := exec.Command("envReader")
  output, _ := child.Output() // Error handling omitted for brevity
  fmt.Println(string(output))
}
```

This produces the following output:

```shell
Before setting special env:
62231: SpecialEnv: <Not Found>

Setting SpecialEnv in parent

After setting special env:
62232: SpecialEnv: SpecialValue
```

In the first invocation, _"SpecialEnv"_ is not set and hence the produced output contains _"\<Not Found\>"_. After this, the parent calls `os.Setenv` and sets a value to it. Note that this value is not exported, so it is _local to the `envParent` processes only_[^1]. Yet, the second invocation of the child process finds it successfully.

This proves that child processes inherit their parent's environment variables.

## Inherited PATH outside the Terminal

During development, it is common to launch Electron apps using the `electron` command from a terminal. This creates a new process and starts the app. Since this process is spawned from a terminal shell, it inherits all the environment variables of the shell session. This includes any custom variables defined in shell config files such as `.bashrc` and `.bash_profile`, like a custom `PATH`.

Once the app is shipped, consumers don't use the terminal to launch it. They use GUI icons, which means the app process is not created by a terminal, but by the Desktop Environment.

The desktop environment process does not concern itself with custom configuration files such as `.bash_profile`, hence its version of the `PATH` environment variable is terse. It does not contain any custom directories the user might have specified in their shell config files.

To show this in action, I wrote a small app with [fyne](https://github.com/fyne-io/fyne) which shows the `PATH` value of the current process. You can find the source [here](https://gist.github.com/haroldadmin/73dc2ca674aaf566c02fe66109b2f859), if interested.

Here's what it looks like when launched from the **GUI**:

![PATH when launched from GUI](/images/finding-right-path/from-gui.png)

And here it is when launched from the **terminal**:

![PATH when launched from Terminal](/images/finding-right-path/from-terminal.png)

Clearly, we're missing out on a lot when our app is launched from the GUI.

(_Also, I really need to cleanup my PATH config_)

## Finding the correct PATH

Fortunately, this problem is easy to solve. With the help of the user's default shell, we can find the complete PATH with the `env` command.

- All processes have access to the `SHELL` environment variable, which contains the location of the default shell executable (eg, _"/bin/bash"_)
- Most shells support the `env` command to list all the environment variables they have access to
- Therefore, running the default shell in an interactive, login mode[^2] with the `env` command should give us, among other variables, the complete PATH.

The Go implementation of this is not too complicated:

```go
func fixPath() {
  // Find the default shell
  defaultShell := os.Getenv("SHELL")

  // Prepare command to get all environment variables
  // eg. /bin/bash -ilc env
  envCommand := exec.Command(defaultShell, "-ilc", "env")
  allEnvVars, _ := envCommand.Output()

  // Find the PATH variable
  for _, envVar := range strings.Split(string(allEnvVars), "\n") {
    if strings.HasPrefix(envVar, "PATH") {
      currentPath := os.Getenv("PATH")
      // Append retrieved PATH to existing value, to get the complete PATH
      completePath := currentPath + string(os.PathListSeparator) + envVar
      // Set the current process's PATH to the complete PATH
      os.Setenv("PATH", completePath)
      return
    }
  }
}
```

I have created a small Go package which do this, and a little more: [_pathfix_](https://github.com/haroldadmin/pathfix).

To use it, add it to your project with `go get github.com/haroldadmin/pathfix`, and then call the `Fix` method to fix your process's PATH:

```go
pathfix.Fix()
```

<br />

Here it is in action, using the app shown earlier.

_Before the fix:_

![Before fix](/images/finding-right-path/before-fix.png)

_After the fix:_

![After fix](/images/finding-right-path/after-fix.png)

If you need to do it in Electron using NodeJS, there is a handy package [_fix-path_](https://github.com/sindresorhus/fix-path) for it.

## Some Notes

- After fixing this problem, I wondered why didn't I encounter this error on Windows. I suspect that Environment Variables on Windows are global by default, and hence every process receives the same `PATH` value every time. I might be wrong about this, please feel free to correct me.

- I also stumbled across PAM Environment Variables which are apparently a more "correct" way of solving this on Linux. You can read more about them in the [documentation](http://www.linux-pam.org/Linux-PAM-html/sag-pam_env.html) and the [Arch Wiki](https://wiki.archlinux.org/index.php/Environment_variables#Using_pam_env). The gist of it is that you need to define your environment variables in a `pam_env` file, but the syntax for it is different. I would be grateful if someone took the time out to explain their usage in a more approachable manner.

- Another potential solution for Linux users is to define their environment variables in the `~/.profile` file, which is sourced by the OS shell on every login. However, just like with the `pam_env` solution, this puts the burden of configuration on your users instead of you.

---

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_

[^1]: Bash environment variables should be declared with `export` to make them available to other processes. eg., `export PATH=$PATH:~/Android/Sdk/platform-tools/`
[^2]: Using the `-il` options when launching the shell makes sure that it reads common config files, such as `*.rc` and `*.profile`. The `-c` at the end just makes sure that the shell reads only the `env` command, and nothing more. Read more about interactive, non-interactive, login and non-login shells [here](https://unix.stackexchange.com/questions/38175/difference-between-login-shell-and-non-login-shell)


Fixing the PATH environment variable inherited by child processes from their parents

Finding the correct PATH



![Hero](/images/androidx-app-startup/hero.png)

The Reddit thread for my blog post on [Exploiting Content Providers](/posts/initialize-using-content-provider) had an interesting comment:

<div className="flex flex-row justify-center">
  <iframe
    id="reddit-embed"
    src="https://www.redditmedia.com/r/androiddev/comments/gll6fu/exploiting_contentprovider_to_initalize_libraries/fqy9xam/?depth=1&amp;showmore=false&amp;embed=true&amp;showmedia=false"
    sandbox="allow-scripts allow-same-origin allow-popups"
    style={{ border: "none" }}
    scrolling="no"
    width="640"
    height="218"
  ></iframe>
</div>

Almost four weeks later, the [AndroidX Startup](https://developer.android.com/topic/libraries/app-startup) library was announced. Following up on [my promise](https://old.reddit.com/r/androiddev/comments/gll6fu/exploiting_contentprovider_to_initalize_libraries/fqyaikf/), let's take a closer look at it.

## The Library

Quoting the docs, here's what App Startup is about:

> The App Startup library provides a straightforward, performant way to initialize components at application startup. Both library developers and app developers can use App Startup to streamline startup sequences and explicitly set the order of initialization.

It promises a few things:

- A simple API which can be used by both libraries and applications
- Performant initialization of components to improve app startup times
- Ability to explicitly request a particular order of initialization

I wanted to try it out in my own project [WhatTheStack](https://github.com/haroldadmin/WhatTheStack), so I decided to take a look at the source code and share my thoughts on it.

### The Idea

A lot of libraries use empty Content Providers to run initialization logic automatically at application startup. As the number of libraries using this trick increases, so does the number of empty content providers. Apparently a large number of them affects startup time significantly, as illustrated by a chart in [this video](https://youtu.be/R3caBPj-6Sg?t=297):

![Cost of any empty Content Provider](/images/androidx-app-startup/cost-content-providers.png)

The idea of the App Startup library is to reduce the number of such content providers by replacing them with just the one supplied by it.

### Usage

App Startup relies on consumers registering the components they want to be initialized at startup. Let's use the canonical example of a custom logger.

To register a component, create an initializer for it by implementing the `Initializer<T>` interface, and add a corresponding Manifest entry:

```kotlin
// LoggerInitializer.kt
class LoggerInitializer : Initializer<MyLogger> {
  override fun create(context: Context): MyLogger {
    MyLogger.init(context)
    return MyLogger.getInstance()
  }

  override fun dependencies(): List<Class<out Initializer<*>>> = emptyList()
}
```

- The `create` method must initialize the component being registered and return an instance of it.
- The `dependencies` method must return a list of `Initializer` classes which need to run before the `create` method of this initializer. This is how you specify the dependencies of the current component to make sure that they are available beforehand.

```xml
// AndroidManifest.xml
<provider
  android:name="androidx.startup.InitializationProvider"
  android:authorities="${applicationId}.androidx-startup"
  android:exported="false"
  tools:node="merge">
  <meta-data  android:name="com.example.LoggerInitializer"
    android:value="androidx.startup" />
</provider>
```

- `<provider>` registers the content provider supplied by App Startup, which is called "_InitializationProvider_".

- `<meta-data>` of the provider registers the initializer we created above.

- Each initializer must go in a separate metadata tag and contain `"androidx.startup"` in the `android:value` attribute. This is important because App Startup uses this string to filter out metadata tags which do not contain initializers.

With this setup complete, App Startup will discover the `LoggerInitializer` class and initialize it automatically at application startup.

#### Lazy Initialization

Components which are discovered through metadata tags are initialized automatically. To disable this for a particular component, add the `tools:node="remove"` attribute to the `<meta-data>` tag for it.

```xml
<provider
  android:name="androidx.startup.InitializationProvider"
  ...>
  <meta-data  android:name="com.example.LoggerInitializer"
    android:value="androidx.startup"
    tools:node="remove" />
</provider>
```

The component can then be lazily initialized using the `AppInitializer` singleton:

```kotlin
AppInitializer
  .getInstance(context)
  .initializeComponent(MyLogger::class.java)
```

To understand why this works, and how App Startup discovers initializers automatically, let's take a look under the hood.

## Under the hood

The source code for App Startup is available [here](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/). This post is based on the first public release (`1.0.0-alpha01`) of the library. There might be major changes as it matures. [This snapshot](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/;drc=ce3f4fbefd5e587269824045a53c622f741ca2e6) represents what the code looks like at the time of writing.

---

_Parts of the code related to tracing, checking for dependency cycles, and handling exceptions have been omitted in the interest of brevity._

---

App Startup makes use of the [Manifest Merger](https://developer.android.com/studio/build/manifest-merge) feature of Android's build tooling. All declarations of the `InitializationProvider` in manifests of dependencies are merged into a single `<provider>` entry in the final manifest of our application. _The final `<provider>` contains `<meta-data>` tags of all merged entries_. On app startup, the OS sees this content provider in the manifest and therefore creates it automatically.

```java
// https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/startup-runtime/src/main/java/androidx/startup/InitializationProvider.java;l=42;drc=2f9dd0f2fe4642438ed3f657dc7f901241a16ca8

public final class InitializationProvider extends ContentProvider {
  @Override
  public boolean onCreate() {
    Context context = getContext();
    if (context != null) {
      AppInitializer.getInstance(context).discoverAndInitialize();
    } else {
      throw new StartupException("Context cannot be null");
    }
  return true;
  }
}
```

InitializationProvider invokes `AppInitializer` upon creation, which then accesses the merged `<meta-data>` tags to find registered initializers.

```java
// https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/startup-runtime/src/main/java/androidx/startup/AppInitializer.java;l=160;drc=564aa855602e82a962ee9b2ca6dff38eaa2ee15e

public final class AppInitializer {
  void discoverAndInitialize() {
    ComponentName provider = new ComponentName(
      mContext.getPackageName(),
      InitializationProvider.class.getName()
    );
    ProviderInfo providerInfo = mContext.getPackageManager()
      .getProviderInfo(provider, GET_META_DATA);
    Bundle metadata = providerInfo.metaData;
    if (metadata != null) {
      // Iterate over all metadata elements
    }
}
```

The metadata bundle is like a `Map` containing merged tags as key-value pairs. In each pair, the key (declared with `android:name`) contains the fully qualified class name of an initializer, and value (`android:value`) contains the string `"androidx.startup"`. Refer to the example shown in the [Usage](#usage) section to see how this is declared.

In each iteration, the key is used to instantiate the Initializer class using Reflection.

```java
// https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/startup-runtime/src/main/java/androidx/startup/AppInitializer.java;l=160;drc=564aa855602e82a962ee9b2ca6dff38eaa2ee15e

void discoverAndInitialize() {
  ...
  Bundle metadata = providerInfo.metaData;
  if (metadata != null) {
    Set<String> keys = metadata.keySet();
    for (String key : keys) {
      String value = metadata.getString(key, null);
      Class<?> clazz = Class.forName(key);
      if (Initializer.class.isAssignableFrom(clazz)) {
        Class<? extends Initializer<?>> component =
                (Class<? extends Initializer<?>>) clazz;
        doInitialize(component, initializing);
      }
    }
  }
}
```

The `doInitialize` method contains the bulk of the code related to component instantiation and initializing its dependencies. The dependencies are initialized before the component itself, repeating this process recursively for each dependency.

```java
// https://cs.android.com/androidx/platform/frameworks/support/+/androidx-master-dev:startup/startup-runtime/src/main/java/androidx/startup/AppInitializer.java;l=105;drc=564aa855602e82a962ee9b2ca6dff38eaa2ee15e

<T> T doInitialize(
  Class<? extends Initializer<?>> component,
  Set<Class<?>> initializing
) {
  Object result;
  if (!mInitialized.containsKey(component)) {
    Object instance = component.getDeclaredConstructor().newInstance();
    Initializer<?> initializer = (Initializer<?>) instance;
    List<Class<? extends Initializer<?>>> dependencies = initializer.dependencies();
    if (!dependencies.isEmpty()) {
      for (Class<? extends Initializer<?>> clazz : dependencies) {
        if (!mInitialized.containsKey(clazz)) {
          doInitialize(clazz, initializing);
        }
      }
    }
    result = initializer.create(mContext);
  } else {
    result = mInitialized.get(component);
  }
  return (T) result;
}
```

After the component is initialized it is stored in a `Map` in `AppInitializer`.

Manifest Merger also explains why we need to add `tools:node="remove"` for lazy initialization to work. A metadata tag containing this attribute is not merged into the final content provider entry of the Manifest, and therefore its initializer is not discovered at startup.

In my opinion, using manifest merger this way is a neat trick. I wonder how many folks outside Google understand the Android build process well enough to come up with unique solutions like this.

## Some Concerns

I like App Startup so far, but I have a few concerns about it in its current state:

- Performance benefit of using App Startup depends on libraries rolling out support for it.
  If a library decides against supporting it, creating initializers for it on your own is futile because doing so will not prevent its content providers from running at startup. You might be able to manually disable them, though.
- Initializers are required to have a no-arg constructor because they need to be instantiated through reflection. This is a drawback if you want to wire them with constructor injection.
- The `create` method of initializers is _required_ to return an instance of the component being initialized. This does not make sense for libraries which are never meant to interacted with in code (such as WhatTheStack) as there is no "instance" to return.
- Someone **will** misuse the `AppInitializer` singleton as a service locator, and that's not what its meant for.

With the negatives out of the way, there is also a lot to like here as well. I am happy to see that libraries do not need to declare App Startup as an `api` dependency in their build config. This means that its usage is opaque to applications consuming those libraries. I also like that now there is a standard way to write initialization code for libraries that run at startup.

Well, not "standard" yet. We will have to wait and see if App Startup takes off.

---

If you would like to see an example of migrating your app/library to use App Startup instead of Content Providers, check out this [PR](https://github.com/haroldadmin/WhatTheStack/pull/5). While you are there, consider checking out some of my other projects on my [GitHub](https://github.com/haroldadmin).

---

_Huge thanks to Subrajyoti Sen and Anubhav Gupta for helping me with this post._

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_


The end of `ContentProvider` exploitation?

Jetpack App Startup: A Deep Dive and Some Concerns

![Hero Image](/images/initialize-using-content-provider/hero.png)

---

**Update: 17/06/2020**

_A month after this blog post, the Jetpack team launched the App Startup library to replace usage of ContentProviders for running init logic at app startup. Read more about it [here](/posts/androidx-app-startup)._

---

Library developers on Android often require applications to initialize their code in the `Application` class. One of my libraries, [WhatTheStack](https://github.com/haroldadmin/WhatTheStack), requires explicit initialization on startup as well:

```kotlin
class MainApplication : Application() {
  override fun onCreate() {
    super.onCreate()
    if (BuildConfig.DEBUG) {
      WhatTheStack(this).init()
    }
  }
}
```

While this approach allows fine grained control over what gets initialized, it has a few drawbacks.

### Bloat

As an application grows, so does the list of things that need to be initialized with it. It is not uncommon to find code like this in Android projects:

```kotlin
class MainApplication : Application() {
  override fun onCreate(){
    super.onCreate()
    Fabric.with(this, new Crashlytics())
    JodaTimeAndroid.init(this)
    Realm.init(this)
    if (BuildConfig.DEBUG) {
      Stetho.initializeWithDefaults(this)
      WhatTheStack(this).init()
    }
  }
}
```

This code snippet is noisy and obscure. It gets progressively difficult to scan it as the list grows. It also makes it easy to initialize something that [shouldn't be](https://twitter.com/Piwai/status/1245524534712602624).

<blockquote className="twitter-tweet">
  <p lang="en" dir="ltr">
    It appears that{" "}
    <a href="https://twitter.com/amazonmusic?ref_src=twsrc%5Etfw">
      @amazonmusic
    </a>{" "}
    is indeed shipping with LeakCanary in production.
    <br />
    <br />I downloaded Amazon Music version 16.9.1 from Play Store and
    decompiled with ShowJava and did find LeakCanary v1.6 in it. The quoted
    tweet shows 2.2 though?{" "}
    <a href="https://t.co/hsGNUHnOE9">https://t.co/hsGNUHnOE9</a>{" "}
    <a href="https://t.co/Qg8DhHPzEd">pic.twitter.com/Qg8DhHPzEd</a>
  </p>
  &mdash; Py ⚔ (@Piwai) <a href="https://twitter.com/Piwai/status/1245524534712602624?ref_src=twsrc%5Etfw">April 2, 2020</a>
</blockquote>
<script
  async
  src="https://platform.twitter.com/widgets.js"
  charSet="utf-8"
></script>

---

### Debug-only dependencies

Some libraries are meant to be included in debug builds only.

Including them in a disabled form in release builds would only increase the application size, or create extra work for the code stripper. For such libraries it is semantically more correct to add them with `debugImplementation` instead of `implementation`. However, there is a catch in doing so:

> _Referencing `debugImplementation` libraries in your application code will result in compilation errors when creating release builds._

This problem has led to the creation of no-op variants of popular libraries[^1]. The build setup with no-op variants requires a different `releaseImplementation` line:

```groovy
debugImplementation "com.facebook.stetho:stetho:1.5.1"
releaseImplementation "net.igenius:stetho-no-op:1.1"
```

While this is not ugly, it certainly is inconvenient. Why do I have to include code that is meant to do nothing just to fix compilation errors?

---

## ContentProvider to the rescue

A [ContentProvider](https://developer.android.com/guide/topics/providers/content-provider-basics) is a core Android component used for making an app's data available to other apps. We shall exploit one of its nifty characteristics:

> An app's ContentProviders are initialized automatically by the system on launch.

To see how this characteristic can be leveraged for auto initialization of our library code, let's look at WhatTheStack's implementation[^2].

```xml

<provider
  android:name=".WhatTheStackInitProvider"
  android:authorities="${applicationId}.WhatTheStackInitProvider"
  android:exported="false"
  android:enabled="true"/>
```

```kotlin
class WhatTheStackInitProvider : ContentProvider() {

  override fun onCreate(): Boolean {
    val applicationContext = context ?: return false
    WhatTheStackInitializer.init(applicationContext)
    return true
  }

  override fun attachInfo(context: Context?, info: ProviderInfo?) {
    super.attachInfo(context, info)
    checkProperAuthority(info)
  }

  private fun checkProperAuthority(info: ProviderInfo?) {
    val contentProviderName = WhatTheStackInitProvider::class.java.name
    require(info?.authority != contentProviderName) {
      "Please provide an applicationId inside your application's build.gradle file"
    }
  }

  // Other overrides
}
```

There is a lot to unpack here, so let's go through it step by step.

- First we need to delcare a `<provider>` element in the library's `AndroidManifest` file.
  The `name` property points to the class which is used as the `ContentProvider`. `authorities` defines a **unique** identifier for it. `exported` is set to `false` as we don't want other applications to use this content provider, and `enabled` property is set to `true` to inform the OS that it can be used.

- Second, we need to define a class extending `ContentProvider`. Upon creation, an instance of this class receives the `onCreate()` callback. This method is supposed to return `true` if the initialization of the content provider is successful, and `false` otherwise. The library is initialized here.

---

**⚠️ Caution! ⚠️**

This callback is run on the main thread of the consuming application. **DO NOT** perform long running operations here.

---

- Next, it receives the `attachInfo()` callback. This contains information about the `ContentProvider` itself, and here we use it to validate the `authority` property[^3].

_... and Voila!_

With this setup in place, our library is initialized automatically, and our consumers do not need to handle it in their application class. Not only that, if our library is meant to be used in debug builds only then the dependency on it can be changed to `debugImplementation` without requiring a no-op variant! 🎉

---

## Disabling automatic initialization

The content provider can be prevented from running at application start by adding the following block in the _application's_ manifest:

```xml
<provider
  android:name="com.haroldadmin.whatthestack.WhatTheStackInitProvider"
  android:authorities="${applicationId}.WhatTheStackInitProvider"
  tools:node="remove" />
```

---

## Exercise caution

A lot of libraries use this approach to hide away init logic, such as [Firebase](https://firebase.googleblog.com/2016/12/how-does-firebase-initialize-on-android.html), [LeakCanary](https://github.com/square/leakcanary/blob/master/leakcanary-object-watcher-android/src/main/java/leakcanary/internal/AppWatcherInstaller.kt), and even [WorkManager](https://developer.android.com/topic/libraries/architecture/workmanager/advanced/custom-configuration#remove-default)

While this is a neat trick, not everyone should rush to update their libraries with automatic initialization. Explicitness and verbosity have their value, and hiding away complexity behind a magic curtain is not always a good idea.

If you are using this trick, make sure your consumers know about it, and give them the option to disable it as well.

_Question, comments or feedback? Feel free to reach out to me on Twitter [@haroldadmin](https://twitter.com/haroldadmin)_

[^1]: The example used is of [Stetho-no-op](https://github.com/iGenius-Srl/stetho-no-op)
[^2]: The PR containing these changes can be found [here](https://github.com/haroldadmin/WhatTheStack/pull/4)
[^3]: This field is created using the `authorities` property in the `AndroidManifest` file. We declare it to be unique by placing the `${applicationId}` suffix on it. In some applications, this ID property is not configured correctly, in which case the `authorities` property uses the library's package name as the fallback. If there are multiple such applications on a user's device, they will all have this content provider with the same value on the authority property, which means it is no longer a unique identifier. Therefore it is necessary to perform this validation.

Running initialization code automatically at application startup

Hi.