Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev #3

Merged
merged 23 commits into from
Sep 7, 2023
Merged

Dev #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9b60811
allow user to set a number of harmonics for each period
bmeyers Aug 30, 2023
8c276a8
new test for having a different number of harmonics in each period
bmeyers Aug 30, 2023
41496c7
regularization matrix function can handle multiple periods and max_cr…
bmeyers Aug 30, 2023
8c55455
installatio instructions
bmeyers Aug 30, 2023
ed1ba2e
cleanup
bmeyers Aug 30, 2023
9ffbf73
code to inject custom basis
bmeyers Aug 30, 2023
dec864b
bug fix
bmeyers Aug 31, 2023
c05bc03
use scipy.sparse.spmatrix
bmeyers Aug 31, 2023
8fe1155
fix test to handle new sparse matrices
bmeyers Aug 31, 2023
c44f6b5
pass through for custom basis
bmeyers Aug 31, 2023
98cf7c1
simplified the logic for generating the regularization weights and pr…
bmeyers Aug 31, 2023
54a7bab
including new arguement
bmeyers Aug 31, 2023
1f8943e
enforce ordering of periods
bmeyers Aug 31, 2023
5cb89bc
make sure to reorder list of harmonics if reording the list of period…
bmeyers Aug 31, 2023
71ae03a
handle custom basis index when reordering periods for user
bmeyers Sep 6, 2023
77b98c4
include missing kwargs
bmeyers Sep 6, 2023
b899d96
fix test
bmeyers Sep 6, 2023
b019b8c
copy user defined bases to extrapolate into the future
bmeyers Sep 6, 2023
a29431f
adding predict function for extrapolating into future time periods
bmeyers Sep 6, 2023
74d58fa
basic forward transform
bmeyers Sep 7, 2023
20d1fae
update notebooks
bmeyers Sep 7, 2023
fbb2ee8
reverse transform working
bmeyers Sep 7, 2023
bdd2229
comments
bmeyers Sep 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,11 @@
# spcqe
Smooth periodic consistent quantile estimation


## Installation

Clone the repository to your local machine and install with pip by navigating to the project directory and running:

```
pip install .
```
15 changes: 4 additions & 11 deletions notebooks/basic_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,14 +232,14 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████| 11/11 [02:22<00:00, 12.92s/it]"
"100%|███████████████████████████████████████████| 11/11 [02:17<00:00, 12.45s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"total time: 142.28 seconds\n"
"total time: 137.13 seconds\n"
]
},
{
Expand All @@ -257,7 +257,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 8,
"metadata": {
"tags": []
},
Expand All @@ -284,7 +284,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 9,
"metadata": {
"tags": []
},
Expand All @@ -308,13 +308,6 @@
"plt.plot(xs, spq.fit_quantiles[slc, :], alpha=0.5)\n",
"plt.xticks(rotation=45);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand Down
56 changes: 25 additions & 31 deletions notebooks/single_quantile-50-sig-decomp.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,27 +181,27 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 12,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<style>#sk-container-id-2 {color: black;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SmoothPeriodicQuantiles(num_harmonics=3, periods=[8760, 168, 24],\n",
" quantiles=array([0.98]), solver=&#x27;osd&#x27;, verbose=True,\n",
" weight=0.1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SmoothPeriodicQuantiles</label><div class=\"sk-toggleable__content\"><pre>SmoothPeriodicQuantiles(num_harmonics=3, periods=[8760, 168, 24],\n",
" quantiles=array([0.98]), solver=&#x27;osd&#x27;, verbose=True,\n",
"<style>#sk-container-id-5 {color: black;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SmoothPeriodicQuantiles(num_harmonics=3, periods=[8760, 168, 24],\n",
" quantiles=array([0.6]), solver=&#x27;osd&#x27;, verbose=True,\n",
" weight=0.1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" checked><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SmoothPeriodicQuantiles</label><div class=\"sk-toggleable__content\"><pre>SmoothPeriodicQuantiles(num_harmonics=3, periods=[8760, 168, 24],\n",
" quantiles=array([0.6]), solver=&#x27;osd&#x27;, verbose=True,\n",
" weight=0.1)</pre></div></div></div></div></div>"
],
"text/plain": [
"SmoothPeriodicQuantiles(num_harmonics=3, periods=[8760, 168, 24],\n",
" quantiles=array([0.98]), solver='osd', verbose=True,\n",
" quantiles=array([0.6]), solver='osd', verbose=True,\n",
" weight=0.1)"
]
},
"execution_count": 6,
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -212,13 +212,13 @@
"P3 = int(24)\n",
"K = 3\n",
"l = 0.1\n",
"spq = SmoothPeriodicQuantiles(K, [P1, P2, P3], quantiles=.98, weight=l, verbose=True, solver='osd')\n",
"spq = SmoothPeriodicQuantiles(K, [P1, P2, P3], quantiles=.60, weight=l, verbose=True, solver='osd')\n",
"spq"
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 13,
"metadata": {
"tags": []
},
Expand All @@ -231,38 +231,32 @@
" QSS: the Quadratic-Separable Solver \n",
" author: Luke Volpatti \n",
"---------------------------------------------------------------\n",
"initial factorization time: 1.04e+00s\n",
"problem size: 70256 variables\n",
"initial factorization time: 1.12e+00s\n",
"(standard algorithm)\n",
"\n",
" ADMM solve \n",
"---------------------------------------------------------------\n",
" iter | objective | primal res | dual res | rho | time (s) \n",
"---------------------------------------------------------------\n",
" 1 | 3.96e+02 1.58e+00 1.07e+00 [0.5 0.02] 3.52e-02\n",
" 25 | 6.71e+02 2.29e-01 6.55e-02 [0.5 0.02] 9.54e-01\n",
" 50 | 6.81e+02 9.21e-02 2.87e-02 [0.5 0.02] 1.93e+00\n",
" 75 | 6.84e+02 4.75e-02 1.62e-02 [0.5 0.02] 2.88e+00\n",
" 100 | 6.84e+02 5.23e-02 1.05e-02 [0.5 0.02] 3.85e+00\n",
" 125 | 6.84e+02 3.13e-02 6.34e-03 [0.5 0.02] 4.80e+00\n",
" 150 | 6.84e+02 2.34e-02 8.24e-03 [0.5 0.02] 5.78e+00\n",
" 175 | 6.84e+02 1.92e-02 3.82e-03 [0.5 0.02] 6.72e+00\n",
" 200 | 6.84e+02 1.46e-02 3.13e-03 [0.5 0.02] 7.71e+00\n",
" 225 | 6.84e+02 1.39e-02 2.78e-03 [0.5 0.02] 8.67e+00\n",
" 250 | 6.84e+02 1.26e-02 2.54e-03 [0.5 0.02] 9.64e+00\n",
" 275 | 6.84e+02 1.06e-02 2.10e-03 [0.5 0.02] 1.06e+01\n",
" 300 | 6.84e+02 9.96e-03 2.00e-03 [0.5 0.02] 1.16e+01\n",
" 325 | 6.84e+02 7.43e-03 1.50e-03 [0.5 0.02] 1.26e+01\n",
" 350 | 6.84e+02 9.06e-03 1.83e-03 [0.5 0.02] 1.35e+01\n",
" 375 | 6.84e+02 5.83e-03 1.19e-03 [0.5 0.02] 1.45e+01\n",
" 380 | 6.84e+02 5.62e-03 1.11e-03 [0.5 0.02] 1.47e+01\n",
" 1 | 8.77e+02 1.65e+00 8.26e-01 [0.5 0.02] 3.66e-02\n",
" 25 | 5.24e+03 7.81e-02 2.39e-02 [0.5 0.02] 9.95e-01\n",
" 50 | 5.26e+03 4.13e-02 1.26e-02 [0.5 0.02] 1.99e+00\n",
" 75 | 5.26e+03 2.85e-02 6.40e-03 [0.5 0.02] 2.96e+00\n",
" 100 | 5.26e+03 1.83e-02 5.21e-03 [0.5 0.02] 3.96e+00\n",
" 125 | 5.26e+03 1.38e-02 2.77e-03 [0.5 0.02] 4.93e+00\n",
" 150 | 5.26e+03 1.00e-02 2.39e-03 [0.5 0.02] 5.93e+00\n",
" 175 | 5.26e+03 6.80e-03 3.70e-03 [0.5 0.02] 6.90e+00\n",
" 200 | 5.26e+03 7.07e-03 1.91e-03 [0.5 0.02] 7.92e+00\n",
" 220 | 5.26e+03 5.52e-03 1.10e-03 [0.5 0.02] 8.72e+00\n",
"---------------------------------------------------------------\n",
"avg time per iter: 3.87e-02s\n",
"avg time per iter: 3.97e-02s\n",
"refactorizations: 0\n",
"total time spent refactorizing: 0.00e+00s\n",
"\n",
"objective value: 683.9867390209845\n",
"total solve time: 15.75s\n",
"total time: 15.92 seconds\n"
"objective value: 5264.373400376573\n",
"total solve time: 9.848s\n",
"total time: 10.02 seconds\n"
]
}
],
Expand Down
Loading