Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40513

SPIP: Support Docker Official Image for Spark

    XMLWordPrintableJSON

Details

    Description

      This SPIP is proposed to add Docker Official Image(DOI) to ensure the Spark Docker images meet the quality standards for Docker images, to provide these Docker images for users who want to use Apache Spark via Docker image.

      There are also several Apache projects that release the Docker Official Images, such as: flink, storm, solr, zookeeper, httpd (with 50M+ to 1B+ download for each). From the huge download statistics, we can see the real demands of users, and from the support of other apache projects, we should also be able to do it.

      After support:

      • The Dockerfile will still be maintained by the Apache Spark community and reviewed by Docker.
      • The images will be maintained by the Docker community to ensure the quality standards for Docker images of the Docker community.

      It will also reduce the extra docker images maintenance effort (such as frequently rebuilding, image security update) of the Apache Spark community.

       

      SPIP DOC: https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o

      DISCUSS: https://lists.apache.org/thread/l1793y5224n8bqkp3s6ltgkykso4htb3

      Attachments

        1.
        		 Add apache/spark-docker repo Sub-task Closed Matei Alexandru Zaharia
        2.
        		 Add .asf.yaml to apache/spark-docker Sub-task Resolved Yikun Jiang
        3.
        		 Add merge_spark_docker_pr.py to help merge commit Sub-task Closed Yikun Jiang
        4.
        		 Add official image dockerfile for Spark v3.3.0 Sub-task Closed Yikun Jiang
        5.
        		 Add PULL_REQUEST_TEMPLATE for spark-docker Sub-task Closed Yikun Jiang
        6.
        		 Make Dockerfile build workflow work in apache repo Sub-task Closed Yikun Jiang
        7.
        		 Enable Spark on K8s integration test for official dockerfiles Sub-task Closed Yikun Jiang
        8.
        		 Add LICENSE and NOTICE for apache/spark-docker Sub-task Closed Yikun Jiang
        9.
        		 Use `spark` username in official image Sub-task Closed Yikun Jiang
        10.
        		 Add dockerfile template Sub-task Closed Yikun Jiang
        11.
        		 Add README for spark-docker Sub-task Closed Yikun Jiang
        12.
        		 Add template support for SPARK_GPG_KEY Sub-task Closed Yikun Jiang
        13.
        		 Cleanup apt lists cache in Dockerfile Sub-task Closed Yikun Jiang
        14.
        		 Rename Check Spark repo as Check Spark Docker repo in GA Sub-task Closed Qian Sun
        15.
        		 Remove pip/setuptools dynamic upgrade Sub-task Closed Yikun Jiang
        16.
        		 Add CONTRIBUTING.md to apache/spark-docker Sub-task Resolved Qian Sun
        17.
        		 Add smoke test in standalone cluster for spark-docker Sub-task Closed Qian Sun
        18.
        		 Add official image dockerfile for Spark v3.3.1 Sub-task Closed Yikun Jiang
        19.
        		 Add "Publish workflow" to help release apache/spark image Sub-task Closed Unassigned
        20.
        		 Move image matrix into version's workflow Sub-task Closed Yikun Jiang
        21.
        		 Add a test workflow to help test image in fork repo Sub-task Closed Yikun Jiang
        22.
        		 Add a script to generate DOI mainifest Sub-task Closed Yikun Jiang
        23.
        		 Add official image Dockerfile for Spark v3.3.2 Sub-task Resolved Yikun Jiang
        24.
        		 Add DOI manifest file for Spark Docker Official Image Sub-task In Progress Yikun Jiang
        25.
        		 Add Spark Docker Official Image doc Sub-task Open Unassigned
        26.
        		 Add doc for Docker Setup in standalone mode Sub-task Open Qian Sun
        27.
        		 Apply entrypoint template change to 3.3.0/3.3.1 Sub-task Closed Yikun Jiang
        28.
        		 Add official image dockerfile for Spark v3.4.0 Sub-task Closed Yikun Jiang
        29.
        		 Add apache/spark docker image overview Sub-task Open Unassigned
        30.
        		 Add Java17 dockerfiles for 3.5.0 Sub-task Resolved Yikun Jiang
        31.
        		 Refactor Dockerfile and workflow based on base image Sub-task Closed Yikun Jiang
        32.
        		 Address DOI comments about /etc/passwd Sub-task Closed Yikun Jiang
        33.
        		 Address comments about /etc/pam.d/su Sub-task Open Unassigned
        34.
        		 Recover sh support in Dockerfiles Sub-task Closed Yikun Jiang
        35.
        		 Switch spark user only when run driver and executor Sub-task Closed Yikun Jiang
        36.
        		 minimize duplication across layers for chmod Sub-task Closed Yikun Jiang
        37.
        		 Use ; instead of && when enable set -ex Sub-task Closed Yikun Jiang
        38.
        		 Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug Sub-task Closed Yikun Jiang
        39.
        		 Add awesome-spark-docker.md Sub-task Closed Yikun Jiang
        40.
        		 Add Apache Spark 3.4.1 Dockerfiles Sub-task Closed Yikun Jiang
        41.
        		 Remove useless lib64 path link in dockerfile Sub-task Closed Unassigned
        42.
        		 Change apt to apt-get and remove useless cleanup Sub-task Closed Yikun Jiang
        43.
        		 Add 'set -eo pipefail' to entrypoint and quote variables Sub-task Closed Yikun Jiang
        44.
        		 Add --batch to gpg command Sub-task Closed Unassigned
        45.
        		 Add official image Dockerfile for Spark 3.3.3 Sub-task Resolved Yuming Wang
        46.
        		 Add official image Dockerfile for Apache Spark 3.5.0 Sub-task Closed Unassigned
        47.
        		 Add Java21 dockerfiles for 3.5.0 Sub-task Open Unassigned

        Activity

          People

            yikunkero Yikun Jiang
            yikunkero Yikun Jiang
            hyukjin.kwon hyukjin.kwon
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: