OpenLI Tutorial 10: Emulating an LEA
You may either watch the tutorial lesson on YouTube by clicking on the imag[e above, or you can download the slides and read them alongside the transcript provided below.
Hello again, and welcome to the tenth chapter of the OpenLI training series. In this lesson, I’m going to show you how to use the libtrace tools to emulate an LEA for testing and debugging purposes. Once you have that up and running, we’ll finally be able to start using the training lab to perform some intercepts!
So why do we need to emulate an LEA? Simply put, you need to be able to confirm that your LI deployment is working correctly and you want to be able to do that before the LEAs request an intercept from you.
A key part of this process will be to conduct tests directly with the LEAs that you will be dealing with, but it is important to note that their time is usually quite limited. Therefore, you want to complete some preliminary testing on your own before approaching the LEAs to test your deployment with them. A mock LEA allows you to do some testing internally, and this means you can resolve any obvious problems with your deployment without wasting anyone else’s time.
And how do you go about emulating an LEA? After all, the equipment that the LEAs use for receiving and decoding intercepts can be even more expensive than the commercial products for performing intercepts.
The good news is that, once again, we have some open source software that can help you out. Libtrace, which you may remember as the library that provides OpenLI with its packet capture capabilities, is also capable of receiving and parsing ETSI handover sessions. This means that any libtrace tool or program can emulate a receiving LEA, so they can act as a receiver for your mediator handovers.
In this case, libtrace is not able to do anything too fancy with the packets it receives via an ETSI handover -- it won’t convert an intercepted voice call into an audio file, for example. But it can do enough that you will be able to test, validate and even debug your OpenLI deployment without needing any outside assistance.
The most useful libtrace tool will be tracepktdump, which you can think of as the libtrace version of the “dump packets to the terminal” feature from tcpdump. Tracepktdump will decode all fields within the TCP/IP headers of a captured packet and print them to your terminal in a nice human readable format.
On top of that, tracepktdump also knows how to decode and display the LI-related fields inside the ETSI headers, so it can be used to validate not just that the correct packets that are being intercepted but also that they have been tagged with the appropriate meta-data required by the ETSI standards.
Let me show you how we would use tracepktdump as a mock agency within our training lab environment.
Of course, the first thing we need to do is get a shell on the agency container. This should be a very familiar process by now, just make sure you run your docker exec command against the “openli-agency” container this time around.
Our emulated LEA will be listening for handover connections from our mediator, so we will once again need to know the IP address of the container so that we can have tracepktdump create a listening socket on the correct interface.
In this case, we want the address of the container at interface eth1, which you may remember from earlier as being on the “openli-agency” network.
In this case, the address is 172.21.0.3 -- it may be slightly different on your container, but as long as the address is on the same subnet as eth2 on your mediator container then everything should be fine.
Each agency handover will require its own TCP port number for the handover session. Remember that there are two handovers to each agency: HI2 for IRI records and HI3 for CC records.
Any port numbers above 1024 should work -- try to choose something memorable though, or at least make a point of writing these numbers down because you’ll need them shortly.
Now that we have an IP address and a port, we can run tracepktdump as a receiver for the HI3 handover, using the command shown on the slide.
tracepktdump etsilive:172.21.0.3:41003
Note the “etsilive” prefix on the URI -- this tells libtrace that you want tracepktdump to act as an ETSI handover endpoint, rather than trying to read packets from a trace file or from a capture interface. This is followed by the IP address we got earlier, then the port number that we chose for HI3. In my case, I’ve chosen 41003.
Once you start tracepktdump successfully, you will not see any output on your terminal -- tracepktdump is now waiting for a mediator to connect to it, which we will configure shortly.
You can stop tracepktdump at any time using Ctrl-C to send an interrupt to the running process.
Since there are two handover interfaces, you’ll need a second tracepktdump instance to receive and parse the records on HI2.
Open up a second terminal, use docker exec to get another shell on your openli-agency container and then run tracepktdump again. This time you’ll want to use the port number you selected for HI2.
Otherwise, the resulting behaviour when you run the command will appear to be the same as it did with the HI3 instance.
Now that our agency is up and running, we now need to tell the OpenLI mediator how to reach it.
For this, we are going to use the OpenLI REST API for the first time. If you are not familiar with a REST API, it is simply a system whereby you can send HTTP requests to a running service to trigger that service to perform certain actions.
In the case of OpenLI, our REST API supports requests that can add, modify, delete or fetch the configuration for the current set of running intercepts, including the agencies that will be receiving those intercepts. Any configuration changes made through the REST API will be immediately perpetuated through the OpenLI components by the provisioner, without any component needing to be restarted or any config files needing to be edited by hand.
You may remember back when we configured the provisioner, we set an IP address and port number for the REST API -- this is where we will be sending our HTTP requests to tell the provisioner about our new mock agency.
Now I'm going to walk you through what the REST API request will look like for adding my mock agency.
Any time you want to add new configuration to OpenLI, you will need to send an HTTP POST request. When adding an agency, the content of that request must be a JSON object and the fields within that object will describe the properties of the agency -- I’ll go through these properties in more detail very shortly.
The POST request must be sent to the URL shown on the slide (http://:/agency) where PROVIP is obviously replaced with your provisioner IP address and RESTAPIPORT is replaced with the port you configured for the REST API a few lessons back. I chose port 8080, so that is what you’ll be seeing on upcoming slides.
There are several properties that need to be specified inside the JSON object when adding a new agency.
The first is the agency ID, which I have set to be “mocklea”. Each agency you add needs to have a unique agency ID, and this is the label that we will use later on to identify which agency should receive any intercepts that we create.
The label is internal to your OpenLI deployment, so feel free to choose whatever makes sense to you. It’s a good idea, though, to make sure that the label is somewhat descriptive of the agency itself just to avoid confusion.
The next two properties are the hi2address and hi3address. These are the IP addresses where the agency is going to be listening for our handover connections.
In our case, these are both going to be the IP address of the openli-agency container that we used when we started up our tracepktdump instances.
Usually, the addresses for HI2 and HI3 will be the same but we’ve implemented them as separate properties just in case you find yourself in a situation where the handovers are split across two hosts.
As you have probably predicted, there are also properties for specifying the port number for each handover, called hi2port and hi3port.
In our case, these take the values of the two port numbers we selected for our mock agency earlier. In a real deployment scenario, the agency will tell you which port numbers you will need to connect to for each handover.
The remaining properties are related to the keep alive requirement specified in the ETSI standards. An agency may choose to mandate that a mediation function must send keep alive messages on a regular basis. This is so that the agency can tell whether your mediator is still functioning, even when there are no intercepted communications to send.
This first property, “keepalivefreq”, will determine how frequently an OpenLI mediator will send keep alive messages to this particular agency. The value for this property is expressed in seconds. Usually, the agency will be able to tell you how frequently they expect keep alives, but otherwise 60 seconds is a reasonable default frequency to choose.
In the rare case that an agency’s LEMF software cannot handle ETSI keep alives, you can set this property to zero to disable keep alives entirely. Do NOT do this unless especially requested by the agency.
LEMFs that receive an ETSI keep alive from a mediator are supposed to reply with a keep alive response message. This allows your mediator to also be able to tell whether the agency’s side of the handover is still functioning. If an agency handover does not reply to a keep alive within a certain time frame, the mediator can assume the handover session has failed and therefore must be re-connected.
The “keepalivewait” property governs how long a mediator should wait for a keep alive response before dropping a handover session. Again, this is expressed in seconds -- I’ve set my timer here for 30 seconds, which I feel is a reasonable default time frame.
Setting this property to zero will prevent the mediator from ever dropping an agency handover due to a failure to respond to a keep alive. This may be necessary for some agencies that you have to deal with, as the keep alive response behaviour seems to be less widely implemented among the vendors that sell to the agencies.
Right, so let’s put it all together and use the REST API to add our mock agency to our OpenLI system.
I’m going to demonstrate this using curl, which is a command line tool for making HTTP queries. curl is good for testing and for making one-off requests, so it is perfect for the purposes of this training lab.
curl will also serve you well when you start putting together your own deployment, although once you have OpenLI in production you may want to integrate the OpenLI REST API with a proper web front-end application.
curl is a very flexible tool, and has an overwhelming amount of capabilities that it presents to the user, but for this exercise we only need to make use of three command line options.
The first is the ‘-X’ option which we use to set the request type to POST.
The second is the ‘-H’ option where we add a content type header to tell the recipient that we are sending JSON in the content.
The third is the ‘-d’ option where we place the JSON object describing our new agency, wrapped in single quotes.
Finally, we end our command with the URL that we want to POST the request to, which we covered earlier. Don’t forget to substitute in the correct IP address and port for your training lab instance!
Once you run your curl command on the provisioner container, you should hopefully see the following HTML content returned as the response to your request.
If not, something went wrong -- the response you get instead will hopefully give you some hints as to what needs to be changed. Keep trying until you get it right!
Now, switch back to the terminals where your tracepktdump instances are running on the openli-agency container.
Each of those instances should now have written to the terminal, the message “Thread 0 is now handling 1 sources”.
Ignoring the lazy grammar, this message indicates that the mediator has managed to successfully establish the handover session with your mock agency.
You should also be able to see evidence of the mock agency announcement in the OpenLI mediator logs, as well as messages that indicate that the handover sessions have been established.
If the handover connection failed for some reason, then you should hopefully see a helpful message explaining why here instead.
And finally, the provisioner logs should also include a log message indicating that the REST API was used to add a new agency.
Of course, things may go wrong and you don’t see some or all of those log messages that indicate success after you run your curl command.
In that case, there are a couple of things to try. First, confirm that you can ping the openli-agency container on its eth1 IP address from the openli-mediator. If the ping fails, then the docker networking for the lab has broken somehow and you may have to rebuild the lab from scratch -- sorry.
If the ping succeeds, then the most likely issue is that you’ve typoed one of the IP addresses or ports for a handover, either when you ran tracepktdump or (more likely) when you made the REST API request.
If you made a typo or mistake in your API request, don’t panic! Just skip ahead to the next slide where I will explain how to delete an agency -- delete the bad one and then try again.
Thankfully, removing an agency via the REST API is very simple.
You just have to send a DELETE request to the REST API service using the following URL path: agency, followed by a slash, followed by the agencyid that you wish to delete.
An example of how I can issue a curl command to delete the agency that I just added earlier is shown on the slide.
If your delete request is successful, you should get the same HTML back in response as you did when you added the agency in the first place.
Querying for the details of a specific agency works in exactly the same way, except you instead send a GET request rather than a DELETE request.
This time, the response that is returned by the REST service will be a JSON object describing the properties of the agency.
The JSON response shown on the slide should look fairly familiar, as it is the exact same JSON object that we sent when we added the agency earlier on!
You can also use a GET request to get the details for all configured agencies, simply by not including a specific agencyid in the URL.
The response will be a list of JSON objects, where each object describes a particular agency.
In the example shown on the slide, I’ve added a second LEA just to demonstrate what a list containing multiple agencies would look like.
Finally, you can use the PUT method to modify the properties of an existing agency. A PUT request looks very similar to the POST request that we used to add a new agency, except our JSON object only needs to contain the properties that are being modified, as well as the agencyid property. Any other properties that are unchanged can be left out of the JSON object being sent and they will remain unchanged.
In this example, I’m using the PUT method to change my HI2 and HI3 ports to different values. Of course, if I do this, then I would need to restart my tracepktdump instances to listen on the new port numbers otherwise the mediator would not be able to establish its handover sessions with my mock agency.
Before I wind this lesson up, I want to just give you a few extra tips that may help you use the libtrace tools more effectively for testing and validating OpenLI.
The first set of tips relate to controlling the large amount of output that can be produced by tracepktdump once you start intercepting real traffic. There are a couple of options that you should be aware of.
The first is that you can use the -c command line flag to tell tracepktdump to halt after it has processed a certain number of packets. This is useful if you just want to quickly look at a few packets and you don’t really care which ones you are shown.
The second tip is that you can pipe the tracepktdump output into a pager utility like less, which will allow you to scroll through the intercepted traffic in the order that it arrives, without worrying about the output exceeding your terminal scroll history. You can also search both forwards and backwards through the output to find packets with specific contents. Instead of piping into less, you can also redirect the tracepktdump output to a file for a more permanent alternative that you can then examine with your preferred paging or editing tool.
For situations where you care less about the specific contents of the intercepted traffic and more about the rate at which traffic is being intercepted, there is a tool called tracertstats which will regularly print packet and byte counts for a live capture format (including an ETSI handover).
Useful command line arguments include the ‘-i’ option which sets the frequency at which the counters are displayed (in seconds) and the ‘-d’ option which enables the display of counters for dropped and missing packets.
The etsilive URI remains the same as it was when we were using tracepktdump to act as the handover endpoint.
There are plenty of other libtrace tools that may be useful, and the full documentation for all libtrace tools can be found at the link shown on the slide (https://github.com/LibtraceTeam/libtrace/wiki/User-Documentation).
And that’s really all you need to know about setting up a mock agency for testing an OpenLI system. We are now finally at the point where we can start to run our first interception in the training lab.
We’re going to start out by intercepting a Voice over IP call, and our next lesson is going to show you how to use the REST API to configure such an intercept.
We’re making some great progress now, so I hope to see you again shortly!