Description
System-of-Systems Data Subject Matter Expert, Cybersecurity Science & Analytics (Azure, Data Lake, Data Engineer, Spark and Hadoop)
Objectives of the role:
- Deliver subject-matter expertise in the use of systems technology in the Microsoft Azure cloud environment, including data & storage architecture, integration, and management
- Design the architecture for our data quality & management framework, to deliver high data quality within our cloud environment while integrating with mandatory data integrity & data quality processes
- Create and maintain data quality standards, including data sourcing and development protocols
- Advise CS&A teams on the development of data models, ETL processes etc that align with our standards and controls
- Support CS&A teams to assess, screen and continuously improve their data quality
- Participate in design review processes to identify & mitigate potential risks
- Define and operate any data governance processes required to ensure that CS&A meets the data quality goals
- Use monitoring & metrics to identify opportunities to improve the CS&A data platforms
- Design and implement solutions to iteratively measure & improve data quality
- Automate data management tasks, using big data tools, AI and custom code
- Contribute to the growth of other specialists though knowledge sharing & consultancy
Required Skills & Experience:
- Experience in similar roles, eg, Data Architect, Lead Data Engineer
- Transmitting complex information to mixed/multiple audiences
- Influencing people, including more senior stakeholders
- Datamodelling tools, techniques & procedures
- Data governance & data management concepts
- Experience working with Big Data environments (TB+ data sets) and tools, eg, Hadoop, Spark etc
- Data quality frameworks, best practices & procedures in an enterprise IT context
- Data catalogue services, eg, Azure Catalog/Purview, Collibra etc
Desirable Skills & Knowledge:
- Data architecture & governance best practices
- Programming tools for data processing, such Python, Scala etc
- Scripting languages for automation, such as PowerShell, Python, etc
- Predictive modelling & machine learning
- Cloud technologies, ideally Azure
- Specific Microsoft Azure services including Azure Data Factory, Azure Databricks, Azure Data Lake Store, Azure Monitor, Azure Log Analytics, Kusto Query Language (KQL)
- Big data tools such as Azure Databricks ML, TensorFlow, Porch etc.
- Data distribution/transport technologies, including ETL, REST, Kafka, Nifi, etc
Industry Experience and Qualifications:
- Candidates will be evaluated primarily upon their ability to demonstrate the competencies required to be successful in the role, as described above. For reference, the typical work experience and educational background of candidates in this role are as follows:
- 2+ years of experience working in a data architecture role
- Bachelors or master's in computer science, engineering, or IT