Showing posts with label HDFS. Show all posts
Showing posts with label HDFS. Show all posts

Tuesday, March 17, 2015

FileCrush replacement format

Error message:
-bash: --replacement=$1-${crush.timestamp}-${crush.task.num}-${crush.file.num}: bad substitution

Fix by using escape character in bash:
--replacement=\$1-\$\{crush.timestamp\}-\$\{crush.task.num\}-\$\{crush.file.num\}

Tuesday, September 9, 2014

Forqlift direct writing to HDFS

1. Combine your existing core-site.xml and hdfs-site.xml to $FORQLIFT_HOME/conf/core-site.xml. Beware of <configuration> tag, it's must be single.
2. Copy following libs from your existing hadoop to $FORQLIFT_HOME/lib.base :
- avro-*.jar
- guava-*.jar
- hadoop-auth-*.jar
- hadoop-common-*.jar
- hadoop-hdfs-*.jar
- protobuf-java-*.jar

Credit to @papaAta