Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turkcell Parser #54

Merged
merged 6 commits into from
Sep 1, 2021
Merged

Turkcell Parser #54

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b3a2682
Turkcell parser.
carbonarok Aug 31, 2021
cb75f52
Modified readme.
carbonarok Aug 31, 2021
7155933
Removed unused imports. Added type.
carbonarok Aug 31, 2021
77b5c24
Travis retry.
carbonarok Aug 31, 2021
042f8ad
Fix mypy.
carbonarok Aug 31, 2021
6ad133a
Moved second table parsing to elif. Added doc string.
carbonarok Sep 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Each provider could use the standard ICal format commented above or define its c
- Lumen
- Megaport
- Telstra
- Turkcell
- Verizon
- Zayo

Expand Down
2 changes: 2 additions & 0 deletions circuit_maintenance_parser/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
PacketFabric,
Telia,
Telstra,
Turkcell,
Verizon,
Zayo,
)
Expand All @@ -31,6 +32,7 @@
PacketFabric,
Telia,
Telstra,
Turkcell,
Verizon,
Zayo,
)
Expand Down
68 changes: 68 additions & 0 deletions circuit_maintenance_parser/parsers/turkcell.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
"""Turkcell parser."""
import logging
import re
from typing import Dict

from bs4.element import ResultSet # type: ignore
from dateutil import parser

from circuit_maintenance_parser.errors import ParsingError
from circuit_maintenance_parser.parser import Html, Impact, CircuitImpact, Status

# pylint: disable=too-many-nested-blocks, too-many-branches

logger = logging.getLogger(__name__)


class HtmlParserTurkcell1(Html):
"""Notifications Parser for Turkcell notifications."""

def parse_html(self, soup, data_base):
"""Execute parsing."""
data = data_base.copy()
try:
self.parse_tables(soup.find_all("table"), data)
return [data]

except Exception as exc:
raise ParsingError from exc

def parse_tables(self, tables: ResultSet, data: Dict):
"""Parse tables."""
glennmatthews marked this conversation as resolved.
Show resolved Hide resolved
# Main table
td_elements = tables[0].find_all("td")
for idx, td_element in enumerate(td_elements):
if "Dear Customer" in td_element.text.strip():
if "planned" in td_element.text.strip():
data["status"] = Status["CONFIRMED"]
else:
data["status"] = Status["CONFIRMED"]
Comment on lines +99 to +102
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have an if/else block that does the same thing in both branches? Is one of these Status values incorrect?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understood that he only found the "planned" use case, and he was enforcing the default one, that is the same.
Ideally, we should have some elif in between when more use cases are added...
is this accurate Kristian?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that is correct Christian. I need more examples with different status. Will be adding more in the future

if "Maintenance Number" in td_element.text.strip():
data["maintenance_id"] = td_elements[idx + 1].text.strip()
elif "Start" in td_element.text.strip():
data["start"] = self.dt2ts(parser.parse(td_elements[idx + 1].text.strip()))
elif "End" in td_element.text.strip():
data["end"] = self.dt2ts(parser.parse(td_elements[idx + 1].text.strip()))
elif "Impact of the maintenance" in td_element.text.strip():
data["summary"] = td_elements[idx + 1].span.text.strip()
if len(tables) == 1:
Copy link
Collaborator

@chadell chadell Aug 31, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if I understand correctly, the Circuit info can be part of the "Impact of the maintenance" section or in a separate table?
maybe, to make this more readable, in this "elif" you could get the p_elements and then implement the if statement at the end:

if len(tables) == 1 and p_elements:
...
elif len(tables) == 2:
...

data["circuits"] = []
p_elements = td_elements[idx + 1].find_all("p")
for element in p_elements:
# Example match:
# Eth-Trunk1.1 up up 111111111111111|01-CUSTOMER|LOCATION|LINK
if re.match(r".+[ \t]([0-1]+\|.+\|.+\|.+)", element.text.strip()):
groups = re.search(r".+[ \t]([0-1]+\|.+\|.+\|.+)", element.text.strip())
details = groups.group(1).split("|")
data["circuits"].append(CircuitImpact(impact=Impact("OUTAGE"), circuit_id=details[0]))
data["account"] = details[1]

# Circuit table
# Possibility that there could be a table inside the first table.
if len(tables) == 2:
tr_elements = tables[1].find_all("tr")
data["circuits"] = []
for tr in tr_elements:
line = tr.text.strip().split("\n\n\n")
data["circuits"].append(CircuitImpact(impact=Impact("OUTAGE"), circuit_id=line[0]))
data["account"] = line[1]
8 changes: 8 additions & 0 deletions circuit_maintenance_parser/providers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
from circuit_maintenance_parser.parsers.lumen import HtmlParserLumen1
from circuit_maintenance_parser.parsers.megaport import HtmlParserMegaport1
from circuit_maintenance_parser.parsers.telstra import HtmlParserTelstra1
from circuit_maintenance_parser.parsers.turkcell import HtmlParserTurkcell1
from circuit_maintenance_parser.parsers.verizon import HtmlParserVerizon1
from circuit_maintenance_parser.parsers.zayo import HtmlParserZayo1

Expand Down Expand Up @@ -161,6 +162,13 @@ class Telstra(GenericProvider):
_default_organizer = "[email protected]"


class Turkcell(GenericProvider):
"""Turkcell provider custom class."""

_parser_classes: Iterable[Type[Parser]] = [HtmlParserTurkcell1]
_default_organizer = "[email protected]"


class Verizon(GenericProvider):
"""Verizon provider custom class."""

Expand Down
126 changes: 126 additions & 0 deletions tests/unit/data/turkcell/turkcell1.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
<div>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<div align="center">
<table style="width:786.45pt;background:#163983;border-collapse:collapse" width="1049" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr style="height:42.85pt">
<td colspan="2" style="width:786.45pt;border:solid #163983 1.0pt;border-bottom:none;padding:2.25pt 2.25pt 0cm 2.25pt;height:42.85pt" width="1049">
<p class="MsoNormal" style="text-align:center" align="center"><span style="font-family:&quot;Century Gothic&quot;,sans-serif;color:white"><img style="width:7.0729in;height:.9791in" id="m_-8407937878423602542m_8660500585316644994_x0000_i1025" src="https://mail.google.com/mail/u/2?ui=2&amp;ik=4ce2f058d0&amp;attid=0.1&amp;permmsgid=msg-f:1708268286934476221&amp;th=17b4fcf4c7cb65bd&amp;view=fimg&amp;sz=s0-l75-ft&amp;attbid=ANGjdJ-110ECCWyvaapsrxStW3lvseDXcI8dmFuwcW-UDtLLbLt3qoxoComaXxtB8zCVjAiAXBe-O2bIpsnBwXAzBeCsrVJ3mqwwhmXD-Wm9Xn5RtgEdwqCcv0HRryg&amp;disp=emb&amp;realattid=17b4fcf00c14ce8e91" alt="Description: Description: Description: cid:[email protected]" data-image-whitelisted="" class="CToWUd" width="679" height="94"><b><u></u><u></u></b></span></p>
<p class="MsoNormal" style="text-align:center;line-height:115%" align="center">
<b><span style="font-size:18.0pt;line-height:115%;font-family:&quot;Century Gothic&quot;,sans-serif;color:white">TURKCELL SUPERONLINE PLANNED MAINTENANCE EMERGENCY<u></u><u></u></span></b></p>
</td>
</tr>
<tr style="height:25.35pt">
<td colspan="2" style="width:786.45pt;border-top:none;border-left:solid #163983 1.0pt;border-bottom:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:25.35pt" width="1049">
<p class="MsoNormal"><span style="font-size:14.0pt"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal" style="line-height:115%"><span style="font-size:14.0pt;line-height:115%">Dear Customer,<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:115%"><span style="font-size:14.0pt;line-height:115%"><br>
We have a planned maintenance on your transmission way.<u></u><u></u></span></p>
<p class="MsoNormal" style="line-height:115%"><span style="font-size:14.0pt;line-height:115%"><u></u>&nbsp;<u></u></span></p>
</td>
</tr>
<tr style="height:11.5pt">
<td style="width:182.1pt;border:none;border-left:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:11.5pt" width="243">
<p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"><span style="font-size:14.0pt;line-height:115%">Maintenance Number:
<u></u><u></u></span></p>
</td>
<td style="width:604.35pt;border:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:11.5pt" width="806" valign="top">
<p class="MsoNormal"><span style="font-size:14.0pt">&nbsp;11111111111111</span><u></u><u></u></p>
</td>
</tr>
<tr style="height:9.75pt">
<td style="width:182.1pt;border:none;border-left:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:9.75pt" width="243">
<p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"><span style="font-size:14.0pt;line-height:115%">Start Tİme:
<u></u><u></u></span></p>
</td>
<td style="width:604.35pt;border:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:9.75pt" width="806" valign="top">
<p class="MsoNormal">&nbsp;07.08.2021 00:00 GMT+3<u></u><u></u></p>
</td>
</tr>
<tr style="height:9.75pt">
<td style="width:182.1pt;border:none;border-left:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:9.75pt" width="243">
<p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"><span style="font-size:14.0pt;line-height:115%">End Tİme:
<u></u><u></u></span></p>
</td>
<td style="width:604.35pt;border:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:9.75pt" width="806" valign="top">
<p class="MsoNormal">&nbsp;07.08.2021 07:00 GMT+3<u></u><u></u></p>
</td>
</tr>
<tr style="height:101.45pt">
<td style="width:182.1pt;border:none;border-left:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:101.45pt" width="243" valign="top">
<p class="MsoNormal" style="margin-bottom:10.0pt;line-height:115%"><span style="font-size:14.0pt;line-height:115%">Impact of the maintenance:<u></u><u></u></span></p>
</td>
<td style="width:604.35pt;border:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:101.45pt" width="806">
<p class="MsoNormal"><span style="font-family:&quot;Century Gothic&quot;,sans-serif">There is a fibercut risk during Sancaktepe routes Circuit transfer work will be carried out at Istanbul Sancaktepe service point. It is foreseen that your services will be interrupted
during the work. </span><span style="font-size:14.0pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><u></u>&nbsp;<u></u></span></p>
<table style="width:623.0pt;border-collapse:collapse" width="831" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr style="height:15.0pt">
<td style="width:361.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="481" valign="bottom" nowrap="">
<p class="MsoNormal">circuit1<wbr>circuit1<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">CUSTOMER<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">Down<u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td style="width:361.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="481" valign="bottom" nowrap="">
<p class="MsoNormal">circuit2<wbr>circuit2<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">CUSTOMER<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">Down<u></u><u></u></p>
</td>
</tr>
<tr style="height:15.0pt">
<td style="width:361.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="481" valign="bottom" nowrap="">
<p class="MsoNormal">circuit3-<wbr>circuit3<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">CUSTOMER<u></u><u></u></p>
</td>
<td style="width:131.0pt;padding:0cm 3.5pt 0cm 3.5pt;height:15.0pt" width="175" valign="bottom" nowrap="">
<p class="MsoNormal">Down<u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:14.0pt"><u></u><u></u></span></p>
</td>
</tr>
<tr style="height:14.65pt">
<td colspan="2" style="width:786.45pt;border-top:none;border-left:solid #163983 1.0pt;border-bottom:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:14.65pt" width="1049">
<p class="MsoNormal" style="margin-right:41.9pt;text-align:justify"><span style="font-size:14.0pt"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal" style="margin-right:41.9pt;text-align:justify"><span style="font-size:14.0pt">Thank you for your cooperation.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:41.9pt;text-align:justify"><span style="font-size:14.0pt">Best regards,<u></u><u></u></span></p>
</td>
</tr>
<tr style="height:15.35pt">
<td colspan="2" style="width:786.45pt;border-top:none;border-left:solid #163983 1.0pt;border-bottom:none;border-right:solid #163983 1.0pt;padding:2.25pt 14.2pt 2.25pt 14.2pt;height:15.35pt" width="1049">
<div class="MsoNormal" style="text-align:center" align="center"><span style="font-family:&quot;Century Gothic&quot;,sans-serif;color:white">
<hr width="100%" size="2" align="center">
</span></div>
<p class="MsoNormal" style="margin-bottom:12.0pt;text-align:center;line-height:115%" align="center">
<span style="font-family:&quot;Century Gothic&quot;,sans-serif;color:white">ino<a href="mailto:[email protected]" target="_blank"><span style="font-family:&quot;Calibri&quot;,sans-serif;color:windowtext;text-decoration:none">[email protected]</span></a> – 0 850 222 46 62<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal"><span style="font-family:&quot;Century Gothic&quot;,sans-serif;color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:white"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u>&nbsp;<u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u>&nbsp;<u></u></span></p>
</div>
29 changes: 29 additions & 0 deletions tests/unit/data/turkcell/turkcell1_result.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
[
{
"account": "CUSTOMER",
"circuits": [
{
"circuit_id": "circuit1circuit1",
"impact": "OUTAGE"
},
{
"circuit_id": "circuit2circuit2",
"impact": "OUTAGE"
},
{
"circuit_id": "circuit3-circuit3",
"impact": "OUTAGE"
}
],
"end": 1625738400,
"maintenance_id": "11111111111111",
"organizer": "[email protected]",
"provider": "turkcell",
"sequence": 1,
"stamp": null,
"start": 1625713200,
"status": "CONFIRMED",
"summary": "There is a fibercut risk during Sancaktepe routes Circuit transfer work will be carried out at Istanbul Sancaktepe service point. It is foreseen that your services will be interrupted\n during the work.",
"uid": "0"
}
]
Loading