From 6GB and 156 million lines down to 921 laws in 13 minutes. Here's how I filtered Lithuania's legal archive.

I continue building a law assistant app and documenting the progress along the way. Now I've decided to get a finite list of law documents. Not to guess which is important and which is not.

## Collecting Data

First challenge was to collect data. I found 6GB of laws. Multiple formats are exposed: HTML, CSV, JSON, JSONL, ASCII, RDF. I chose CSV.

I'm parsing this CSV line by line. Chunking legal documents by CSV header with id and document type.

The solution is to read the document line by line (hundreds of millions of lines) and chunk by metadata header. Metadata header contains "id" and multiple other attributes, not always containing values.

With the data structure understood, the next challenge was determining what to keep and what to discard.

## Filtering

This archive contains multiple legal documents. I need only important ones. For now I want laws only. I do not want amendments (duplications in actual redaction of law), regulations, decrees, resolutions, orders, protocols, international law ratifications. Only laws. And law is a form of a document. Ratification is a law legalizing international law. So I need only local laws for now.

Once I knew what to filter, I needed to extract consistent metadata from inconsistent sources.

## Normalizing Data

For a legal document, I want to have document name, document type, and effective date at least. Spoiler alert: metadata does not contain this data in most cases. And document type is my internal type for filtering.

With clarity on what data I needed, the bottleneck became how fast I could process it.

## Performance

In parsing 6GB of data, performance is important. Whether parsing takes 10 minutes or 100 minutes is a big difference at the prototype level. After normalizing, filtering, and refactoring, I ended up with 30 seconds per million lines. And I have hundreds of millions of lines (156 million to be precise).

Quick win for me was to ask an LLM to inspect node performance snapshot:

- `node --prof <your command>` creates v8 performance logs

- `node --prof-process` transforms these logs to readable profiler snapshot

Then the LLM found hotspots and optimized those. The key optimization was using a bitmask (similar to a Bloom filter) for fast document type filtering. Instead of string comparisons for every line, I create a character mask:

```typescript
function maskForLetters(value: string): number {
  let mask = 0;
  for (let i = 0; i < value.length; i += 1) {
    const code = value.charCodeAt(i);
    if (code >= 97 && code <= 122) {
      mask |= 1 << (code - 97);
    } else if (code >= 65 && code <= 90) {
      mask |= 1 << (code - 65);
    }
  }
  return mask;
}
```



Then filter lines with a simple bitwise operation:

typescript

```typescript
if ((lineMask & type.firstCharMask) === 0) continue;
```

This reduced processing time from 30 seconds to 5 seconds per million lines. Total processing time for 156 million lines: around 13 minutes.

## Summary

As a result, I have 921 "important" laws extracted, with document name, effective date, and a link to the archive. The next step is defining a data model and parsing the important laws.

Building a Lithuanian Law Assistant with LLM and RAG (part 2)

I'm building another one, a new app. This time I'll share my progress and findings with you. I'm building a Lithuanian law assistant with LLM using RAG. As nowadays usual, it is a chat interface which must be aware of multiple Lithuanian law entities and relationships between them to answer questions: how is this law act's paragraph applied in practice? What are the consequences? What is the process? I genuinely believe that this application will be useful and the application of LLM + RAG is a good fit here (let me know if you think the opposite).

## Tech Stack

- PostgreSQL + Prisma

- Node.js + NestJS

- Vite, React (Radix UI components)

- Vitest, pnpm, TypeScript

- Models: text-embedding-3-small, gpt-4o-mini

I am using NestJS for the first time. First impression: this is a Java Spring framework implemented in TypeScript. It has a rich ecosystem, and for now it is a good fit for a small project.

## What I've Done So Far

I have a chat interface in dark theme with calming green elements. I haven't encountered any complexity with the UI at all. It is simple and it has one screen.

I've learned the ingestion pipeline meaning. Very important part of the application. One of the most important. What is ingestion? This is a process of transforming any data to structured data with embeddings (vectors) for later retrieval. The quality and accuracy of responses depends on the quality of data. For example, you want to ingest labor law. Then you need to find one, export it as any type of artifact (docx, pdf, html), parse it to text (I've chosen markdown), and then chunk it to entities (act → article → paragraph), create embeddings for those chunks and save embeddings to the database.

### Parsing and Chunking

Currently parsing and chunking are the most time-consuming tasks for me. Current implementation can ingest any Lithuanian act (act → article → paragraph). This step requires thorough testing and multiple iterations. At this point regular expressions are my solution for parsing and chunking.

### Conversation

For conversation I'm doing the following:

- Combining history to one prompt - the tradeoff: multiple LLM calls and longer response time, but more accurate context augmentation

- Get embeddings for this prompt

- Query my chunks by embedding - I store only paragraph chunks, because article chunks may be too long, so I've omitted embedding article as a whole. Though I store reference to an article

- Query LLM with chat history (last 12 messages) + chunks from DB - not sure that 12 last messages is a good solution here. But I need to make a decision how to prevent context overflow

For now this process works as expected for chat interface. Response time is acceptable, but database and servers are deployed locally. Deployed application latency may add some milliseconds. At this point it is fast enough.

## What Are My Next Steps?

I will work on expanding the domain, adding legal documents, procedures, institutions, decisions, concepts, principles and consequences. This is not the hardest "business" domain. But for me there is a lot what I should learn first.

Adding new entities to the application requires:

- Understanding why it is important

- Understanding what this entity does

- Parsing and chunking data

- Storing and linking to existing domain entities