Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Identity provisioning fails to add subject DN to CSR using EST and x509 authentication when using identity certificate #6579

Closed
cookieofcode opened this issue Aug 9, 2022 · 21 comments

Comments

@cookieofcode
Copy link

Expected Behavior

The configured Distinguished Name (DN) in the "subject" configuration should be added to the Certificate Signing Request (CSR) using Enrollment over Secure Transport (EST) when using X509 (client) authentication with an identity certificate.

Current Behavior

Only the common name of the subject is included in the CSR of the identity certificate. Other fields such as "organization_unit" (OU) are ommited. The subject configuration is included for the EST bootstrap identity ("bootstrap_identity"), but not for the EST device identity ("identity").

Intercepted EST network traffic (aziot-certd) for "identity_cert":

POST [https://<...>/.well-known/est/<...>/simpleenroll HTTP/1.1](https://<...>/.well-known/est/<...>/simpleenroll)
content-type: application/pkcs10
content-transfer-encoding: base64
host: <...>
content-length: 825

Decoded Payload (openssl req -in csr -inform der -noout -text):
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = GW-01-OP-001
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus: <...>
                Exponent: <...>
        Attributes:
            <...>
    Signature Algorithm: sha256WithRSAEncryption
         <...>
Intercepted network traffic for "bootstrap_identity"
POST [https://<...>/.well-known/est/<...>/simpleenroll HTTP/1.1](https://<...>/.well-known/est/<...>/simpleenroll)
content-type: application/pkcs10
content-transfer-encoding: base64
host: <...>
content-length: 825

Decoded Payload (openssl req -in csr -inform der -noout -text):
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = GW-01-OP-001, OU = GW-01
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    <...>
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        Attributes:
        Requested Extensions:
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
    Signature Algorithm: ecdsa-with-SHA256
         <...>

The EST server is configured to only accept CSR that contains an "organization_unit". This results in the error: {"errors":[{"code":"invalid_input","message":"Please provide value for subject.organization_unit"}]}. This field is a constraint for an operational certificate by the PKI / EST server provider "DigiCert IoT Manger" (referenced in the documentation, see https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-configure-est-server?view=iotedge-2020-11#est-server)

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. Install IoT Edge according to the documentation (see https://docs.microsoft.com/en-us/azure/iot-edge/how-to-provision-single-device-linux-symmetric?view=iotedge-2020-11&tabs=azure-portal%2Cubuntu#install-iot-edge)
  2. Configure DPS provisioning with EST (see configuration in Context (Environment)) using "subject" instead of "common_name" for the "identity" (see template.toml#L180, #5742).

Context (Environment)

Configuration:

hostname = "ubuntu"
trust_bundle_cert = "file:///<...>/root.pem"

[cert_issuance.est]
trusted_certs = [
     "file:///<...>/cacerts.pem",
]

[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net/"
id_scope = "<...>"

[provisioning.attestation]
method = "x509"
registration_id = "GW-01-OP-001"

[provisioning.attestation.identity_cert]
method = "est"
subject = { CN = "GW-01-OP-001", OU = "GW-01" }
url = "https://<...>/.well-known/est/<...>"
identity_cert = "file:///<...>/identity_cert.pem"
identity_pk = "file:///<...>/identity_key.pem"

Output of iotedge check

Click here
Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ production readiness: identity certificates expiry - OK
√ production readiness: EST identity and bootstrap certificates expiry - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (TimedOut) : Operation timed out
    One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
24 check(s) succeeded.
7 check(s) raised warnings. Re-run with --verbose for more details.
1 check(s) raised errors. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

  • Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]: Ubuntu 20.04.04 LTS
  • Architecture [e.g. amd64, arm32, arm64]: amd64

Runtime Versions

  • aziot-edged [run iotedge version]: 1.3.0
  • Edge Agent [image tag (e.g. 1.0.0)]: 1.3.0
  • Edge Hub [image tag (e.g. 1.0.0)]: 1.3.0
  • Docker/Moby [run docker version]: 20.10.17+azure-1

Logs

aziot-edged logs
Aug 09 01:07:57 ubuntu systemd[1]: Started Azure IoT Identity Service.
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [INFO] - Starting service...
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [INFO] - Version - 1.3.0
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [INFO] - Provisioning starting. Reason: Startup
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 75 bytes
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", keyd.sock)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 1 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is empty
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 357 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 1 headers
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is empty
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [INFO] - !!! parameter "id" has an invalid value
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [INFO] - !!! caused by: not found
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [INFO] - --> 400 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 190 bytes
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 81 bytes
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (73 bytes)
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", certd.sock)
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 369 bytes
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is empty
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", keyd.sock)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- DELETE /keypair?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "248"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 204 {}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 83 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (56 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /keypair?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "56"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 174 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 357 bytes
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", keyd.sock)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 123 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 465 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 124 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 123 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 465 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (248 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 124 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (355 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "355", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 470 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 1080 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 1 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is empty
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- GET /keypair/est-id-device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 365 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (256 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "256", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 123 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (955 bytes)
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [INFO] - <-- POST /certificates?api-version=2020-09-01 {"content-type": "application/json", "host": "certd.sock", "content-length": "955"}
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 82 bytes
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (256 bytes)
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", keyd.sock)
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::connect::http] connecting to 192.168.117.1:8888
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::connect::http] connecting to 192.168.117.1:8888
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (256 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "256", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 465 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (256 bytes)
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "256", "content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::io] flushed 124 bytes
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-keyd[15506]: 2022-08-09T08:07:57Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::connect::http] connected to 192.168.117.1:8888
Aug 09 01:07:57 ubuntu aziot-certd[15512]: 2022-08-09T08:07:57Z [DBUG] - [hyper::client::connect::http] connected to 192.168.117.1:8888
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::io] flushed 1043 bytes
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::io] flushed 122 bytes
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::io] parsed 12 headers
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::conn] incoming body is chunked encoding
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::decode] incoming chunked header: 0x64 (100 bytes)
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::client::pool] pooling idle connection for ("https", clientauth.demo.one.digicert.com)
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [ERR!] - !!! internal error
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [ERR!] - !!! caused by: could not create cert
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [ERR!] - !!! caused by: EST endpoint did not return successful response: 400 Bad Request b"{\"errors\":[{\"code\":\"invalid_input\",\"message\":\"Please provide value for subject.organization_unit\"}]}"
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [INFO] - --> 500 {"content-type": "application/json"}
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::io] flushed 155 bytes
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::io] parsed 3 headers
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (28 bytes)
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [DBUG] - [hyper::client::pool] pooling idle connection for ("http", certd.sock)
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: internal error
Aug 09 01:07:58 ubuntu aziot-certd[15512]: 2022-08-09T08:07:58Z [DBUG] - [hyper::proto::h1::conn] read eof
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] - service encountered an error
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] - caused by: internal error
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] - caused by: could not create certificate
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] - caused by: internal error
Aug 09 01:07:58 ubuntu aziot-identityd[15898]: 2022-08-09T08:07:58Z [ERR!] -    0: <unknown>
Aug 09 01:07:58 ubuntu aziot-identityd[15898]:    1: <unknown>
Aug 09 01:07:58 ubuntu systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Aug 09 01:07:58 ubuntu systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

@nyanzebra
Copy link
Contributor

Attempting to reproduce the situation described. Will update with findings after testing DPS with EST and x509.

@nyanzebra
Copy link
Contributor

From speaking internally, DPS only checks CN for provisioning, so it generally does not matter that we drop the other DN fields for the identity cert. This also appears to be a known issue that we are tracking internally to fix. In the meantime, if you don't require EST you can follow this: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-x509?view=iotedge-2020-11&tabs=individual-enrollment%2Cubuntu.

@cookieofcode
Copy link
Author

cookieofcode commented Aug 11, 2022

Thank you for your response @nyanzebra. The problem described above does not lie within DPS (or the device connection group), this works fine. It is a technical limitation of the current configuration of the EST Server (DigiCert IoT Device Manager), that the organization unit must be included in the subject of the CSR otherwise it will not be signed.

This limitation is because we're using X509 authentication to the EST Server (with the manufacturing/birth certificate) to obtain an operational certificate. Another field than the common name (in the subject DN) is then used by IoT Device Manager to match both certificates to the same device.

@nyanzebra
Copy link
Contributor

@varunpuranik, should I mark this as bug? Also, do we have any alternative EST configurations that we can suggest?

@jlian
Copy link
Member

jlian commented Aug 17, 2022

It is a technical limitation of the current configuration of the EST Server (DigiCert IoT Device Manager), that the organization unit must be included in the subject of the CSR otherwise it will not be signed.

Is it possible to configure it to remove this constraint?

@cookieofcode
Copy link
Author

It is a technical limitation of the current configuration of the EST Server (DigiCert IoT Device Manager), that the organization unit must be included in the subject of the CSR otherwise it will not be signed.

Is it possible to configure it to remove this constraint?

@jlian We're in contact with the provider and still waiting for a response. Currently, we are not able to remove this constraint and we're required to adjust the subject in the EST request. This results in a workaround which can not be used in production. Do you have an estimate about the timeline when a fix could be included in a LTS release?

@jlian
Copy link
Member

jlian commented Sep 15, 2022

We've triaged the bug and are looking into a fix, but not too sure that it'll make the next release (1.4.2) which needs to go out in time to take the latest .NET security patch and dependency updates. My guess would be that it'll take about a month for this fix to be released as part of 1.4.3. Do you need it sooner?

@pebneter
Copy link

Hi @jlian , I'm working on the same project as @cookieofcode .
If we can have a fix in November, this would still be ok for us. We are not in production yet but want to advance with our prototype.

@jlian
Copy link
Member

jlian commented Sep 28, 2022

We've picked up the work and are currently forecasting late Oct to release this in 1.4.3.

@jlian
Copy link
Member

jlian commented Oct 5, 2022

Still targeting late Oct to release in 1.4.3.

CC Azure/iot-identity-service#455

@MZDN
Copy link

MZDN commented Oct 28, 2022

I am facing similar issue, when I trying to provision a device using EST server (libest from cisco) installed outside the machine where azure iot identity service (aziotctl 1.4.1 ) runs. When debugging the network traffic, I could see that the aziotctl sends the device cert request(CSR) but est-server was not able to validate the request. does it mean that the csr created by aziotctl is invalid?
Please note that everything worked for me when i am having est-server and aziotctl(1.4.1) on the same vm.

@jlian
Copy link
Member

jlian commented Nov 2, 2022

Hey folks, the change is merged (thanks @onalante-msft).

Ideally, we could have you try it before we take it for the release:

  1. Download the patched binary here https://github.com/Azure/iot-identity-service/actions/runs/3364093065
  2. And then manually install following these steps https://azure.github.io/iot-identity-service/installation.html

@cookieofcode, @pebneter, and @MZDN do you think you could give it a try this week?

@pebneter
Copy link

pebneter commented Nov 3, 2022

H @jlian, thanks for the update!
Unfortunately, @cookieofcode is out of office for a while and I'm not sure how to reproduce it.

@MZDN
Copy link

MZDN commented Nov 7, 2022

Hi @jlian,
I gave a try, but unfortunately it does not fix. Please notice that the provided package version is 1.4.0.
On the EST-server, I could see following in the log at each request from aziotctl:
OSSL error: 139832112584448:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42
This error means that:
42 | bad_certificate | A certificate was corrupt, contained signatures that did not verify correctly, etc.

@jlian
Copy link
Member

jlian commented Nov 7, 2022

@MZDN I think your issue is a bit different compared to the original, which is about the missing Subject DN fields such as the OU. AFAIK, the original issue affects IoT Edge using DigiCert IoT Manger EST provider.

Since you're trying this with libest, I believe the issue is different.

Could you please open a separate issue and fill out the bug report, so that we could properly diagnose? In it, please also include if you're following steps from https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-configure-est-server (which uses libest) and if there are deviations.

@MZDN
Copy link

MZDN commented Nov 11, 2022

Hi @jlian,
I opened a new issue Azure/iot-identity-service#483. I think it is more related to iot-identity-service project.
Thanks

@cookieofcode
Copy link
Author

cookieofcode commented Dec 8, 2022

Hey folks, the change is merged (thanks @onalante-msft).

Ideally, we could have you try it before we take it for the release:

1. Download the patched binary here https://github.com/Azure/iot-identity-service/actions/runs/3364093065

2. And then manually install following these steps https://azure.github.io/iot-identity-service/installation.html

@cookieofcode, @pebneter, and @MZDN do you think you could give it a try this week?

Hi @jlian. Thank you for the patch.

I've tried the patch with the following versions:

ubuntu@ubuntu:/var/lib/aziot$ sudo dpkg -l | grep aziot
ii  aziot-edge                                 1.4.0-1                               amd64        Azure IoT Edge Module Runtime
ii  aziot-identity-service                     1.4.0~dev-1                           amd64        Azure IoT Identity Service and related services

And the following configuration:

[provisioning.attestation]
method = "x509"
registration_id = "GW-01-OP-001"

[provisioning.attestation.identity_cert]
method = "est"
subject = { CN = "GW-01-OP-001", OU = "GW-01" }
url = "https://<...>/.well-known/est/<...>"
identity_cert = "file:///<...>/identity_cert.pem"
identity_pk = "file:///<...>/identity_key.pem"

Unfortunately, the intercepted call to the EST endpoint does not contain another field in the subject than the common name (the organizational unit is still missing).

In the mean time, we found a solution with the EST provider to omit the organizational unit in the certificate signing request. Regardless of this, it should be possible to configure an alternative subject in the certificate signing request.

@jlian
Copy link
Member

jlian commented Dec 21, 2022

@cookieofcode that's weird, I wonder what's the difference between your setup vs Azure/iot-identity-service#455 (comment)?

@onalante-msft any ideas?

@onalante-msft
Copy link
Contributor

@cookieofcode Have you re-run iotedge config apply? It is required to propagate the augmented attestation information to identityd.

@github-actions
Copy link

This issue is being marked as stale because it has been open for 30 days with no activity.

@jlian
Copy link
Member

jlian commented Feb 1, 2023

IoT Edge 1.4.8 is released and includes this fix (which is part of identity service 1.4.2)

@jlian jlian closed this as completed Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants