Skip to main content

Metadata Change Log Event V1

Event Type

MetadataChangeLog_v1

Overview

This event is emitted when any aspect on DataHub Metadata Graph is changed. This includes creates, updates, and removals of both "versioned" aspects and "time-series" aspects.

Disclaimer: This event is quite powerful, but also quite low-level. Because it exposes the underlying metadata model directly, it is subject to more frequent structural and semantic changes than the higher level Entity Change Event. We recommend using that event instead to achieve your use case when possible.

Event Structure

The fields include

NameTypeDescriptionOptional
entityUrnStringThe unique identifier for the Entity being changed. For example, a Dataset's urn.False
entityTypeStringThe type of the entity being changed. Supported values include dataset, chart, dashboard, dataFlow (Pipeline), dataJob (Task), domain, tag, glossaryTerm, corpGroup, & corpUser.False
entityKeyAspectObjectThe key struct of the entity that was changed. Only present if the Metadata Change Proposal contained the raw key struct.True
changeTypeStringThe change type. UPSERT or DELETE are currently supported.False
aspectNameStringThe entity aspect which was changed.False
aspectObjectThe new aspect value. Null if the aspect was deleted.True
aspect.contentTypeStringThe serialization type of the aspect itself. The only supported value is application/json.False
aspect.valueStringThe serialized aspect. This is a JSON-serialized representing the aspect document originally defined in PDL. See https://github.com/datahub-project/datahub/tree/master/metadata-models/src/main/pegasus/com/linkedin for more.False
previousAspectValueObjectThe previous aspect value. Null if the aspect did not exist previously.True
previousAspectValue.contentTypeStringThe serialization type of the aspect itself. The only supported value is application/jsonFalse
previousAspectValue.valueStringThe serialized aspect. This is a JSON-serialized representing the aspect document originally defined in PDL. See https://github.com/datahub-project/datahub/tree/master/metadata-models/src/main/pegasus/com/linkedin for more.False
systemMetadataObjectThe new system metadata. This includes the the ingestion run-id, model registry and more. For the full structure, see https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/mxe/SystemMetadata.pdlTrue
previousSystemMetadataObjectThe previous system metadata. This includes the the ingestion run-id, model registry and more. For the full structure, see https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/mxe/SystemMetadata.pdlTrue
createdObjectAudit stamp about who triggered the Metadata Change and when.False
created.timeNumberThe timestamp in milliseconds when the aspect change occurred.False
created.actorStringThe URN of the actor (e.g. corpuser) that triggered the change.

Sample Events

Tag Change Event

{
"entityType": "container",
"entityUrn": "urn:li:container:DATABASE",
"entityKeyAspect": null,
"changeType": "UPSERT",
"aspectName": "globalTags",
"aspect": {
"value": "{\"tags\":[{\"tag\":\"urn:li:tag:pii\"}]}",
"contentType": "application/json"
},
"systemMetadata": {
"lastObserved": 1651516475595,
"runId": "no-run-id-provided",
"registryName": "unknownRegistry",
"registryVersion": "0.0.0.0-dev",
"properties": null
},
"previousAspectValue": null,
"previousSystemMetadata": null,
"created": {
"time": 1651516475594,
"actor": "urn:li:corpuser:datahub",
"impersonator": null
}
}

Glossary Term Change Event

{
"entityType": "dataset",
"entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)",
"entityKeyAspect": null,
"changeType": "UPSERT",
"aspectName": "glossaryTerms",
"aspect": {
"value": "{\"auditStamp\":{\"actor\":\"urn:li:corpuser:datahub\",\"time\":1651516599479},\"terms\":[{\"urn\":\"urn:li:glossaryTerm:CustomerAccount\"}]}",
"contentType": "application/json"
},
"systemMetadata": {
"lastObserved": 1651516599486,
"runId": "no-run-id-provided",
"registryName": "unknownRegistry",
"registryVersion": "0.0.0.0-dev",
"properties": null
},
"previousAspectValue": null,
"previousSystemMetadata": null,
"created": {
"time": 1651516599480,
"actor": "urn:li:corpuser:datahub",
"impersonator": null
}
}

Owner Change Event

{
"auditHeader": null,
"entityType": "dataset",
"entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)",
"entityKeyAspect": null,
"changeType": "UPSERT",
"aspectName": "ownership",
"aspect": {
"value": "{\"owners\":[{\"type\":\"DATAOWNER\",\"owner\":\"urn:li:corpuser:datahub\"}],\"lastModified\":{\"actor\":\"urn:li:corpuser:datahub\",\"time\":1651516640488}}",
"contentType": "application/json"
},
"systemMetadata": {
"lastObserved": 1651516640493,
"runId": "no-run-id-provided",
"registryName": "unknownRegistry",
"registryVersion": "0.0.0.0-dev",
"properties": null
},
"previousAspectValue": {
"value": "{\"owners\":[{\"owner\":\"urn:li:corpuser:jdoe\",\"type\":\"DATAOWNER\"},{\"owner\":\"urn:li:corpuser:datahub\",\"type\":\"DATAOWNER\"}],\"lastModified\":{\"actor\":\"urn:li:corpuser:jdoe\",\"time\":1581407189000}}",
"contentType": "application/json"
},
"previousSystemMetadata": {
"lastObserved": 1651516415088,
"runId": "file-2022_05_02-11_33_35",
"registryName": null,
"registryVersion": null,
"properties": null
},
"created": {
"time": 1651516640490,
"actor": "urn:li:corpuser:datahub",
"impersonator": null
}
}

FAQ

Where can I find all the aspects and their schemas?

Great Question! All MetadataChangeLog events are based on the Metadata Model which is comprised of Entities, Aspects, and Relationships which make up an enterprise Metadata Graph. We recommend checking out the following resources to learn more about this:

You can also find a comprehensive list of Entities + Aspects of the Metadata Model under the Metadata Modeling > Entities section of the official DataHub docs.