Skip to content

Dirac v6r10

Jump to bottom Edit New page
andresailer edited this page Oct 22, 2014 · 15 revisions

The are 3 main features introduced within this release:

  1. The full use of the new version of the Request Management System, introduced first within release v6r9, together with the Transformation System and Data Management system.

  2. Many changes are applied in the TransformationSystem, and particularly in the TransformationDB.

  3. A new "package", named Workflow, has been introduced. This package makes it easier, for VOs, to define Job Workflows, introducing generic modules implementations that can be extended by the VOs.

Those new features that need special care when upgrading from previous versions to v6r10 are outlined below.

New RMS everywhere

Using the new RMS system everywhere means using it for all the replication activities, that were using the old RMS system up to version v6r9. The old system, populating the old set of tables, will not be feed any more. System administrators might want to check that all the requests are treated within the old system before stopping the corresponding agents: this is the reason why, in v6r10 code (as well as v6r9), a mix of services and agents from the old and new system are present. The code treating the old system will be removed from the next minor, or major, release of DIRAC. If you are upgrading from DIRAC v6r9, no new agents nor services need to be started.

Changes have been applied in several places, including Accounting and Transformation System.

FTSDB

As part of the changes for the new RMS, a new DB, and a new set of tables will be created at the start of the FTSManagerService, provided that the FTSDB has been previously created. This DB, together with the new implementation of the service and clients, will substitute the old system. Old requests present in the old system will keep to be treated by the old services and agents.

ReqDB

Apply the following:

ALTER Table Operation MODIFY Arguments MEDIUMBLOB;

Accounting

Changes at the DB level to clean up indexes and speed up queries. The following changes could be done in the mysql database independently from the release installed:

  • drop of duplicate indexes on ac_key* tables (valueindex)

  • drop of duplicate indexes on ac_in* tables (idIndex)

  • creation of multi-field index on ac_bucket* tables (UniqueIndex). Fields are reordered wrt UniqueConstraint index, beginning with startTime

  • drop of (all) unused single-field indexes on ac_bucket* tables

  • drop of less performant multi-field index on ac_bucket* tables (UniqueConstraint)

  • change the following record::

    update ac_catalog_Types set keyFields='Status,Site,User,UserGroup,JobGroup,MinorStatus,ApplicationStatus,JobSplitType' where name='Dirac-Production_WMSHistory';

    here DIRAC-Production is the DIRAC setup. Do that for all the setups.

As an example, here is the SQL for WMSHistory:

CREATE UNIQUE INDEX UniqueIndex ON ac_bucket_LHCb-Production_WMSHistory(startTime,Status,Site,User,UserGroup,JobGroup,MinorStatus,ApplicationStatus,JobSplitType,bucketLength);

DROP INDEX MinorStatusIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX UserGroupIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX JobGroupIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX UserIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX JobSplitTypeIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX ApplicationStatusIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX SiteIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX StatusIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX startTimeIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX bucketLengthIndex ON ac_bucket_LHCb-Production_WMSHistory;

DROP INDEX UniqueConstraint ON ac_bucket_LHCb-Production_WMSHistory;

Please note: create and drop index operations may take from minutes to several hours (even days), depending on table size

Transformation System

Few code changes are:

  • the TransformationClient now provides an easily extensible State Machine for Transformation Status and for the status of the Transformation Files.
  • when deriving a transformation, the Transformation Files status will be deterministic.
  • the methods for the Replicas table have been deleted. But the biggest changes are at the DB level: please read further.

TransformationDB

The tables definition is switching from MyISAM to InnoDB, and few triggers are being introduced. The code will be fully compatible, so there is no need, for any VOs, to switch MyISAM to InnoDB. It is anyway highly recommended. Applying this change needs special care. It has to be noted that, for this release, both the tables definition, and the triggers, are defined in the TransformationDB.sql file, and that system administrators are required to manually intervene at the database level to apply them. The changes to apply follow:

  • TransformationDB.Replicas can be fully removed, including its content.
  • The other changes are coded in https://github.com/DIRACGrid/DIRAC/pull/1731/files, and includes settings of PKs, FKs, and a trigger. Pay special attention to the TransformationFiles and TransformationTasks table.

In any case, to repeat: VOs can schedule the change of the schema at their favourite time, independently from the introduction of DIRAC v6r10, because tables defined within MyISAM will keep working.

After the tables were changed, restart the TransformationManager.

Workflow

The new workflow package provides four generic modules. Detailed instructions on how to use it and extend it can be found in the DIRAC.Workflow.Modules.ModuleBase module.

ResourceStatus

Some new policies have been introduced and are ready to be used. The configuration of the authorization should be changed from "get = all" to "select = all".

WorkloadManagementSystem

Add a custom footer
Add a custom sidebar
Clone this wiki locally