비 클라우드 환경의 GCS 커넥터

Aug 17 2020

hadoop 3 버전의 GCS 커넥터를 설치하고 Install.md에 설명 된대로 core-site.xml에 아래 구성을 추가했습니다 . 의도는 로컬 클러스터의 hdfs에서 클라우드 스토리지로 데이터를 마이그레이션하는 것입니다.

core-site.xml

fs.gs.project.id=<project-id>
fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
google.cloud.auth.service.account.enable=true
google.cloud.auth.service.account.json.keyfile=<path to key file>

서비스를 다시 시작했습니다.

파일을 나열하기 위해 클라우드의 버킷에 액세스하려고하면 실패합니다.

 hdfs --loglevel TRACE dfs -ls gs://data-store/

    20/08/17 15:44:09 DEBUG gcs.GoogleHadoopFileSystemBase: GHFS version: hadoop3-2.1.4
    20/08/17 15:44:09 DEBUG fs.FileSystem: gs:// = class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/lib/gcs-connector-hadoop3-latest.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: file:// = class org.apache.hadoop.fs.LocalFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: har:// = class org.apache.hadoop.fs.HarFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /usr/hdp/3.0.0.0-1634/hadoop-hdfs/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /usr/hdp/3.0.0.0-1634/hadoop-hdfs/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /usr/hdp/3.0.0.0-1634/hadoop-hdfs/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: s3n:// = class org.apache.hadoop.fs.s3native.NativeS3FileSystem from /usr/hdp/3.0.0.0-1634/hadoop-mapreduce/hadoop-aws-3.1.0.3.0.0.0-1634.jar
    20/08/17 15:44:09 DEBUG fs.FileSystem: Looking for FS supporting gs
    20/08/17 15:44:09 DEBUG fs.FileSystem: looking for configuration option fs.gs.impl
    20/08/17 15:44:09 DEBUG fs.FileSystem: Filesystem gs defined in configuration option
    20/08/17 15:44:09 DEBUG fs.FileSystem: FS for gs is class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
    20/08/17 15:44:09 DEBUG gcs.GoogleHadoopFileSystemBase: initialize(path: gs://data-store/, config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, initSuperclass: true)
    20/08/17 15:44:09 DEBUG gcs.GoogleHadoopFileSystemBase: initializeDelegationTokenSupport(config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, path: gs://data-store/)
    20/08/17 15:44:09 TRACE gcs.GoogleHadoopFileSystemBase: Failed to initialize delegation token support
    java.lang.IllegalStateException: Delegation Tokens are not configured
            at com.google.cloud.hadoop.repackaged.gcs.com.google.common.base.Preconditions.checkState(Preconditions.java:508)
            at com.google.cloud.hadoop.fs.gcs.auth.GcsDelegationTokens.init(GcsDelegationTokens.java:65)
            at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initializeDelegationTokenSupport(GoogleHadoopFileSystemBase.java:578)
            at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:555)
            at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:510)
            at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
            at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
            at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
            at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
            at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
            at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
            at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249)
            at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232)
            at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
            at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
            at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
            at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
    20/08/17 15:44:09 DEBUG gcs.GoogleHadoopFileSystemBase: GHFS_ID=GHFS/hadoop3-2.1.4: configure(config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml)

구성과 관련하여 놓친 것이 있는지 확실하지 않습니다. 클러스터가 Kerberos 화되었으며 유효한 Kerberos 티켓이 있습니다 (이 시나리오에서 관련성이 있는지 확실하지 않음).

구성에 빠진 것이 있습니까? 어떤 제안?

답변

2 cyxxy Aug 18 2020 at 05:42

스택 추적 Delegation Tokens are not configured은 실제로 붉은 청어입니다. 여기 에서 GCS 커넥터 코드를 읽으면 커넥터가 항상 위임 토큰 지원 구성을 시도하지만 fs.gs.delegation.token.binding구성을 통해 바인딩을 지정하지 않으면 실패하지만 추적에 표시되는 예외는 삼켜집니다.