CSCFAIRMETA-762-New-links-for-SYKE-datasets (#644)

* CSCFAIRMETA-647: [FIX] bug with almost simultaneous requests

- LightFileserializer made modifications to class variable
  so when two requests came almost at the same time the latter
  was using modified version of the variable. Fixed with deepcopy
- Changed app to use local Elasticsearch with tests since it is now
  working

* CSCFAIRMETA-710: [FIX] _mapping endpoint and ES in testcases

- Correctly call elasticsearch when request comes to _mapping endpoint
- Tests can be run against local elasticsearch which reduces the
  population time
- Follow naming convention in refdata loader with non-internal
  functions

* CSCFAIRMETA-710: [FIX] remove unnecessary variable

* CSCFAIRADM-337: [ADD] Update domain references (#631)

- These mostly concern test data, but should be updated still

* CSCFAIRMETA-704 [FIX] merge-draft-does-not-save-changes
- In merge_draft take preservation_state from draft cr

* CSCFAIRMETA-704-merge-draft-does-not-save-changes
- Change metax-demo to metax.demo

* CSCFAIRMETA-716: [FIX] create-draft-internal-error-when-draft-already-exists
- Add a check for next_draft

* CSCFAIRMETA-712-Issues-with-emails
-Add migration to change email and organization name

* CSCFAIRMETA-712-Issues-with-emails
- Add check whether metadata_owner_org or metadata_provider_org exists

* CSCFAIRMETA-636-return-if-files-are-part-of-some-dataset
- Add ?keys=datasets|files and ?keysonly=bool
- Remove parameter ?detailed as it is the same as ?keys=files

* CSCFAIRMETA-636-return-if-files-are-part-of-some-dataset
- Add test for parameter ?keysonly

* CSCFAIRMETA-636-return-if-files-are-part-of-some-dataset
- Add support for leaving argument ?detailed

* CSCFAIRMETA-636-return-if-files-are-part-of-some-dataset
Add ?keysonly return input list except standalone files or datasets without files

* CSCFAIRMETA-636-return-if-files-are-part-of-some-dataset
- Refactor sql's
- Fix parameters 'request' and 'preferred_identifier'

* CSCFAIRMETA-762-New-links-for-SYKE-datasets
- Save old notation into own field
- Replace notation with new guid
- Add datasets folder into migrations
- Replace syke url for harvester

Co-authored-by: Tommi Pulli <[email protected]>
Co-authored-by: Tommi Pulli <[email protected]>
Co-authored-by: Mattias Levlin <[email protected]>

src/metax_api/api/oaipmh/base/metax_oai_server.py

@@ -17,7 +17,8 @@
 from metax_api.services import CatalogRecordService as CRS
 from metax_api.services.datacite_service import DataciteException, convert_cr_to_datacite_cr_json
 
-SYKE_URL_PREFIX_TEMPLATE = 'http://metatieto.ymparisto.fi:8080/geoportal/catalog/search/resource/details.page?uuid=%s'
+# SYKE_IDENTIFIER_PREFIX
+SYKE_URL_PREFIX_TEMPLATE = 'https://metadata.ymparisto.fi/dataset/%s'
 DATACATALOGS_SET = 'datacatalogs'
 DATASETS_SET = 'datasets'
 OAI_DC_MDPREFIX = 'oai_dc'

src/metax_api/migrations/0027_auto_20201013_1404.py

@@ -0,0 +1,56 @@
+# Generated by Django 3.0.7 on 2020-10-13 11:04
+
+from django.db import migrations
+import csv
+import json
+import logging
+
+logger = logging.getLogger(__name__)
+
+def rename(apps, schema_editor):
+    CatalogRecord = apps.get_model('metax_api', 'CatalogRecord')
+
+    # Loop through csv
+    with open('metax_api/migrations/datasets/SYKE_csv.csv', newline='') as csvfile:
+        reader = csv.DictReader(csvfile, delimiter=";", quotechar='"')
+
+        for row in reader:
+            old_guid = row["vanha guid"]
+
+            if old_guid == 'vanha guid': # Check if is heade value
+                pass
+            else:
+                # Get dataset
+                try:
+                    cr = CatalogRecord.objects.get(research_dataset__other_identifier__0__notation = old_guid)
+                    new_guid = row["uusi guid"]
+                    cr.research_dataset['other_identifier'][0]['notation'] = new_guid
+                    cr.research_dataset['other_identifier'][0]['old_notation'] = old_guid
+                    cr.user_modified = "migration_0027"
+                    cr.save()
+                    logger.info(f"successful migration of cr {cr.id} with new guid {new_guid} and old guid {old_guid}")
+                except CatalogRecord.DoesNotExist:
+                    pass
+
+def revert(apps, schema_editor):
+    CatalogRecord = apps.get_model('metax_api', 'CatalogRecord')
+    modified = CatalogRecord.objects.filter(user_modified="migration_0027")
+    for cr in modified:
+        try:
+            o_ids = cr.research_dataset['other_identifier']
+            if "old_notation" in o_ids[0].keys():
+                cr.research_dataset['other_identifier'][0]['notation'] = o_ids[0]["old_notation"]
+                del cr.research_dataset['other_identifier'][0]['old_notation']
+                cr.save()
+        except CatalogRecord.DoesNotExist:
+            pass
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('metax_api', '0026_auto_20200917_1615'),
+    ]
+
+    operations = [
+        migrations.RunPython(rename, revert),
+    ]