Merge pull request #92 from networktocode/develop

Version v2.0.3

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## v2.0.3 - 2021-10-01
+
+### Added
+
+- #84 - New parser added for text. Added new provider `AWS` using `Text` and `EmailSubjectParser`
+- #91 - `Provider` now adds `_include_filter` and `_exclude_filter` attributes (using regex) to filter in and out notifications that are relevant to be parsed vs other that are not, avoiding false positives.
+
+### Fixed
+
+- #90 - Improved handling of Lumen scheduled maintenance notices
+
 ## v2.0.2 - 2021-09-28
 
 ### Fixed

README.md

@@ -19,15 +19,31 @@ during a NANOG meeting that aimed to promote the usage of the iCalendar format.
 proposed iCalendar format, the parser is straight-forward and there is no need to define custom logic, but this library
 enables supporting other providers that are not using this proposed practice, getting the same outcome.
 
-You can leverage on this library in your automation framework to process circuit maintenance notifications, and use the standarised output to handle your received circuit maintenance notifications in a simple way.
+You can leverage this library in your automation framework to process circuit maintenance notifications, and use the standardized [`Maintenance`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) to handle your received circuit maintenance notifications in a simple way. Every `maintenance` object contains, at least, the following attributes:
+
+- **provider**: identifies the provider of the service that is the subject of the maintenance notification.
+- **account**: identifies an account associated with the service that is the subject of the maintenance notification.
+- **maintenance_id**: contains text that uniquely identifies the maintenance that is the subject of the notification.
+- **circuits**: list of circuits affected by the maintenance notification and their specific impact.
+- **status**: defines the overall status or confirmation for the maintenance.
+- **start**: timestamp that defines the start date of the maintenance in GMT.
+- **end**: timestamp that defines the end date of the maintenance in GMT.
+- **stamp**: timestamp that defines the update date of the maintenance in GMT.
+- **organizer**: defines the contact information included in the original notification.
+
+> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to more details about these attributes.
 
 ## Workflow
 
 1. We instantiate a `Provider`, directly or via the `init_provider` method, that depending on the selected type will return the corresponding instance.
-2. Each `Provider` have already defined multiple `Processors` that will be used to get the `Maintenances` when the `Provider.get_maintenances(data)` method is called.
-3. Each `Processor` class can have a pre defined logic to combine the data extracted from the notifications and create the final `Maintenance` object, and receives a `List` of multiple `Parsers` that will be to `parse` each type of data.
-4. Each `Parser` class supports one or more data types and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant key/values.
-5. When calling the `Provider.get_maintenances(data)`, the `data` argument is an instance of `NotificationData` (which is just a collection of multiple `DataParts`, each one with a `type` and a `content`) that will be used by the corresponding `Parser` when the `Processor` will try to match them.
+2. Get an instance of the `NotificationData` class. This instance groups together `DataParts` which each contain some content and a specific type (that will match a specific `Parser`). For example, a `NotificationData` might describe a received email message, with `DataParts` corresponding to the subject line and body of the email. There are factory methods to initialize a `NotificationData` describing a single chunk of binary data, as well as others to initialize one directly from a raw email message or `email.message.EmailMessage` instance.
+3. Each `Provider` uses one or more `Processors` that will be used to build `Maintenances` when the `Provider.get_maintenances(data)` method is called.
+4. Each `Processor` class uses one or more `Parsers` to process each type of data that it handles. It can have custom logic to combine the parsed data from multiple `Parsers` to create the final `Maintenance` object.
+5. Each `Parser` class supports one or a set of related data types, and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant keys/values.
+
+<p align="center">
+<img src="https://raw.githubusercontent.com/nautobot/nautobot-plugin-circuit-maintenance/develop/docs/images/new_workflow.png" width="800" class="center">
+</p>
 
 By default, there is a `GenericProvider` that support a `SimpleProcessor` using the standard `ICal` `Parser`, being the easiest path to start using the library in case the provider uses the reference iCalendar standard.
 
@@ -43,6 +59,7 @@ By default, there is a `GenericProvider` that support a `SimpleProcessor` using
 
 #### Supported providers based on other parsers
 
+- AWS
 - AquaComms
 - Cogent
 - Colt
@@ -65,15 +82,39 @@ By default, there is a `GenericProvider` that support a `SimpleProcessor` using
 The library is available as a Python package in pypi and can be installed with pip:
 `pip install circuit-maintenance-parser`
 
-## Usage
+## How to use it?
 
-> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to understand the meaning
-> of the output attributes.
+The library requires two things:
 
-## Python Library
+- The `notificationdata`: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).
+- The `provider` identifier: used to select the proper `Provider` which contains the `processor` logic to take the proper `Parsers` and use the data that they extract. By default, the `GenericProvider` (used when no other provider type is defined) will support parsing of `iCalendar` notifications using the recommended format.
+
+### Python Library
+
+First step is to define the `Provider` that we will use to parse the notifications. As commented, there is a `GenericProvider` that implements the gold standard format and can be reused for any notification matching the expectations.
 
 ```python
-from circuit_maintenance_parser import init_provider, NotificationData
+from circuit_maintenance_parser import init_provider
+
+generic_provider = init_provider()
+
+type(generic_provider)
+<class 'circuit_maintenance_parser.provider.GenericProvider'>
+```
+
+However, usually some `Providers` don't fully implement the standard and maybe some information is missing, for example the `organizer` email or maybe a custom logic to combine information is required, so we allow custom `Providers`:
+
+```python
+ntt_provider = init_provider("ntt")
+
+type(ntt_provider)
+<class 'circuit_maintenance_parser.provider.NTT'>
+```
+
+Once we have the `Provider` ready, we need to initialize the data to process, we call it `NotificationData` and can be initialized from a simple content and type or from more complex structures, such as an email.
+
+```python
+from circuit_maintenance_parser import NotificationData
 
 raw_data = b"""BEGIN:VCALENDAR
 VERSION:2.0
@@ -97,93 +138,63 @@ END:VEVENT
 END:VCALENDAR
 """
 
-ntt_provider = init_provider("ntt")
-
 data_to_process = NotificationData.init_from_raw("ical", raw_data)
 
-maintenances = ntt_provider.get_maintenances(data_to_process)
+type(data_to_process)
+<class 'circuit_maintenance_parser.data.NotificationData'>
+```
+
+Finally, with we retrieve the maintenances (it is a `List` because a notification can contain multiple maintenances) from the data calling the `get_maintenances` method from the `Provider` instance:
+
+```python
+maintenances = generic_provider.get_maintenances(data_to_process)
 
 print(maintenances[0].to_json())
 {
-  "account": "137.035999173",
-  "circuits": [
-    {
-      "circuit_id": "acme-widgets-as-a-service",
-      "impact": "NO-IMPACT"
-    },
-    {
-      "circuit_id": "acme-widgets-as-a-service-2",
-      "impact": "OUTAGE"
-    }
-  ],
-  "end": 1444471200,
-  "maintenance_id": "WorkOrder-31415",
-  "organizer": "mailto:[email protected]",
-  "provider": "example.com",
-  "sequence": 1,
-  "stamp": 1444435800,
-  "start": 1444464000,
-  "status": "TENTATIVE",
-  "summary": "Maint Note Example",
-  "uid": "42"
+"account": "137.035999173",
+"circuits": [
+{
+"circuit_id": "acme-widgets-as-a-service",
+"impact": "NO-IMPACT"
+},
+{
+"circuit_id": "acme-widgets-as-a-service-2",
+"impact": "OUTAGE"
+}
+],
+"end": 1444471200,
+"maintenance_id": "WorkOrder-31415",
+"organizer": "mailto:[email protected]",
+"provider": "example.com",
+"sequence": 1,
+"stamp": 1444435800,
+"start": 1444464000,
+"status": "TENTATIVE",
+"summary": "Maint Note Example",
+"uid": "42"
 }
 ```
 
-## CLI
+Notice that, either with the `GenericProvider` or `NTT` provider, we get the same result from the same data, because they are using exactly the same `Processor` and `Parser`. The only difference is that `NTT` notifications come without `organizer` and `provider` in the notification, and this info is fulfilled with some default values for the `Provider`, but in this case the original notification contains all the necessary information, so the defaults are not used.
 
-```bash
-$ circuit-maintenance-parser --data-file tests/unit/data/ical/ical1 --data-type ical
-Circuit Maintenance Notification #0
-{
-  "account": "137.035999173",
-  "circuits": [
-    {
-      "circuit_id": "acme-widgets-as-a-service",
-      "impact": "NO-IMPACT"
-    }
-  ],
-  "end": 1444471200,
-  "maintenance_id": "WorkOrder-31415",
-  "organizer": "mailto:[email protected]",
-  "provider": "example.com",
-  "sequence": 1,
-  "stamp": 1444435800,
-  "start": 1444464000,
-  "status": "TENTATIVE",
-  "summary": "Maint Note Example",
-  "uid": "42"
-}
+```python
+ntt_maintenances = ntt_provider.get_maintenances(data_to_process)
+assert maintenances_ntt == maintenances
 ```
 
-```bash
-$ circuit-maintenance-parser --data-file tests/unit/data/zayo/zayo1.html --data-type html --provider-type zayo
-Circuit Maintenance Notification #0
-{
-  "account": "clientX",
-  "circuits": [
-    {
-      "circuit_id": "/OGYX/000000/ /ZYO /",
-      "impact": "OUTAGE"
-    }
-  ],
-  "end": 1601035200,
-  "maintenance_id": "TTN-00000000",
-  "organizer": "[email protected]",
-  "provider": "zayo",
-  "sequence": 1,
-  "stamp": 1599436800,
-  "start": 1601017200,
-  "status": "CONFIRMED",
-  "summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span",
-  "uid": "0"
-}
-```
+### CLI
+
+There is also a `cli` entrypoint `circuit-maintenance-parser` which offers easy access to the library using few arguments:
+
+- `data-file`: file storing the notification.
+- `data-type`: `ical`, `html` or `email`, depending on the data type.
+- `provider-type`: to choose the right `Provider`. If empty, the `GenericProvider` is used.
 
 ```bash
 circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo
 Circuit Maintenance Notification #0
 {
-  "account": "Linode",
+  "account": "some account",
   "circuits": [
     {
       "circuit_id": "/OGYX/000000/ /ZYO /",
@@ -203,6 +214,68 @@ Circuit Maintenance Notification #0
 }
 ```
 
+## How to Extend the Library?
+
+Even though the library aims to include support for as many providers as possible, it's likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new `Provider` is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.
+
+First step is creating a new file: `circuit_maintenance_parser/parsers/abcde.py`. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from `circuit_maintenance_parser.parser`. In the example, we only need to import `Html` and in the child class implement the methods required by the class, in this case `parse_html()` which will return a `dict` with all the data that this `Parser` can extract. In this case we have to helper methods, `_parse_bs` and `_parse_tables` that implement the logic to navigate the notification data.
+
+```python
+from typing import Dict
+import bs4  # type: ignore
+from bs4.element import ResultSet  # type: ignore
+from circuit_maintenance_parser.parser import Html
+
+class HtmlParserABCDE1(Html):
+    def parse_html(self, soup: ResultSet) -> Dict:
+        data = {}
+        self._parse_bs(soup.find_all("b"), data)
+        self._parse_tables(soup.find_all("table"), data)
+        return [data]
+
+    def _parse_bs(self, btags: ResultSet, data: Dict):
+      ...
+
+    def _parse_tables(self, tables: ResultSet, data: Dict):
+      ...
+```
+
+Next step is to create the new `Provider` by defining a new class in `circuit_maintenance_parser/provider.py`. This class that inherits from `GenericProvider` only needs to define two attributes:
+
+- `_processors`: is a `list` of `Processor` instances that uses several data `Parsers`. In this example, we don't need to create a new custom `Processor` because the combined logic serves well (the most likely case), and we only need to use the new defined `HtmlParserABCDE1` and also the generic `EmailDateParser` that extract the email date. Also notice that you could have multiple `Processors` with different `Parsers` in this list, supporting several formats.
+- `_default_organizer`: this is a default helper to fill the `organizer` attribute in the `Maintenance` if the information is not part of the original notification.
+
+```python
+class ABCDE(GenericProvider):
+    _processors: List[GenericProcessor] = [
+        CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]),
+    ]
+    _default_organizer = "[email protected]"
+```
+
+And expose the new `Provider` in `circuit_maintenance_parser/__init__.py`:
+
+```python
+from .provider import (
+    GenericProvider,
+    ABCDE,
+    ...
+)
+
+SUPPORTED_PROVIDERS = (
+    GenericProvider,
+    ABCDE,
+    ...
+)
+```
+
+Last, but not least, you should update the tests!
+
+- Test the new `Parser` in `tests/unit/test_parsers.py`
+- Test the new `Provider` logic in `tests/unit/test_e2e.py`
+
+... adding the necessary data samples in `tests/unit/data/abcde/`.
+
 # Contributing
 
 Pull requests are welcomed and automatically built and tested against multiple versions of Python through Travis CI.
@@ -225,6 +298,7 @@ The project is following Network to Code software development guidelines and is
 1. Define the `Parsers`(inheriting from some of the generic `Parsers` or a new one) that will extract the data from the notification, that could contain itself multiple `DataParts`. The `data_type` of the `Parser` and the `DataPart` have to match. The custom `Parsers` will be placed in the `parsers` folder.
 2. Update the `unit/test_parsers.py` with the new parsers, providing some data to test and validate the extracted data.
 3. Define a new `Provider` inheriting from the `GenericProvider`, defining the `Processors` and the respective `Parsers` to be used. Maybe you can reuse some of the generic `Processors` or maybe you will need to create a custom one. If this is the case, place it in the `processors` folder.
+   - The `Provider` also supports the definition of a `_include_filter` and a `_exclude_filter` to limit the notifications that are actually processed, avoiding false positive errors for notification that are not relevant.
 4. Update the `unit/test_e2e.py` with the new provider, providing some data to test and validate the final `Maintenances` created.
 5. **Expose the new `Provider` class** updating the map `SUPPORTED_PROVIDERS` in `circuit_maintenance_parser/__init__.py` to officially expose the `Provider`.
 

circuit_maintenance_parser/__init__.py

@@ -7,6 +7,7 @@
 from .provider import (
     GenericProvider,
     AquaComms,
+    AWS,
     Cogent,
     Colt,
     EUNetworks,
@@ -29,6 +30,7 @@
 SUPPORTED_PROVIDERS = (
     GenericProvider,
     AquaComms,
+    AWS,
     Cogent,
     Colt,
     EUNetworks,

circuit_maintenance_parser/constants.py

@@ -0,0 +1,4 @@
+"""Constants used in the library."""
+
+EMAIL_HEADER_SUBJECT = "email-header-subject"
+EMAIL_HEADER_DATE = "email-header-date"

circuit_maintenance_parser/data.py

@@ -4,6 +4,8 @@
 
 import email
 from pydantic import BaseModel, Extra
+from circuit_maintenance_parser.constants import EMAIL_HEADER_SUBJECT, EMAIL_HEADER_DATE
+
 
 logger = logging.getLogger(__name__)
 
@@ -73,9 +75,8 @@ def init_from_emailmessage(cls: Type["NotificationData"], email_message) -> Opti
             cls.walk_email(email_message, data_parts)
 
             # Adding extra headers that are interesting to be parsed
-            data_parts.add(DataPart("email-header-subject", email_message["Subject"].encode()))
-            # TODO: Date could be used to extend the "Stamp" time of a notification when not available, but we need a parser
-            data_parts.add(DataPart("email-header-date", email_message["Date"].encode()))
+            data_parts.add(DataPart(EMAIL_HEADER_SUBJECT, email_message["Subject"].encode()))
+            data_parts.add(DataPart(EMAIL_HEADER_DATE, email_message["Date"].encode()))
             return cls(data_parts=list(data_parts))
         except Exception:  # pylint: disable=broad-except
             logger.exception("Error found initializing data from email message: %s", email_message)