We are doing the data transformations using Google Dataproc and all our data is residing in Dataproc Hive tables. How do i transfer/move this data to BigQuery.Answer1:
Transfer to BigQuery from Hive seems to have a standard pattern:<ul><li>dump your Hive into Avro files</li> <li>Load those files in BigQuery</li> </ul>
See an example here: <a href="https://stackoverflow.com/questions/46958916/migrate-hive-table-to-google-bigquery/47038501#47038501" rel="nofollow">Migrate hive table to Google BigQuery</a>
As mentioned above, take care about the types compatibility between Hive/Avro/BigQuery.
And for the first time I guess it would not hurt to do some validations by comparing that the tables on both Hive and BigQuery have the same data: <a href="https://github.com/bolcom/hive_compared_bq" rel="nofollow">https://github.com/bolcom/hive_compared_bq</a>