# Dataproc

> Run Dataproc locally with LocalCloud: connection settings, supported workflows, known limits, and code examples.
>
> Source: https://local.cloud/services/dataproc/

Dataproc provides managed Spark and Hadoop execution on Google Cloud. LocalCloud runs supported cluster jobs and serverless batches through local Docker runtimes.

## Typical uses

Use when a pipeline uses Spark or Hadoop to transform partitioned datasets, and you need to submit jobs or serverless batches through the Dataproc API.

Examples

- Spark ETL and data transformations
- Hadoop and batch-processing tests

Connection

Environment variable

`DATAPROC_EMULATOR_HOST=localhost:5380`

Runtime details

- **Endpoints:** gRPC :5380
- **Minimum tier:** Community
- **Persistence:** metadata
- **Implementation:** Local Facade

## Example usage

Service ID: `dataproc`. Starts by default. Use an already running LocalCloud runtime for the examples below.

Export the local project and mapped endpoints in your client process, then use the client settings below. [First-time setup](https://local.cloud/docs/)

```bash
# Export GOOGLE_CLOUD_PROJECT and DATAPROC_EMULATOR_HOST
eval "$(localcloud env)"
```

REST gateway examples use the generated HTTP address in `SERVICE_USAGE_EMULATOR_HOST`; native SDKs can use a different protocol.

Java · REST:

### Java · REST

### List submitted jobs

List submitted jobs; Dataproc runs Spark and Hadoop transformations over your datasets.

Call the Google Cloud REST API with the JDK HTTP client. Requires JDK 17+. Results reflect resources already created in your local project.

Discovery check · reads existing local state.

#### Save as Hello.java

```java
import java.net.URI;
import java.net.http.*;
import java.time.Duration;
import java.util.Objects;

public class Hello {
  public static void main(String[] args) throws Exception {
    var project = Objects.requireNonNull(System.getenv("GOOGLE_CLOUD_PROJECT"));
    var endpoint = Objects.requireNonNull(System.getenv("SERVICE_USAGE_EMULATOR_HOST"));
    var url = URI.create(endpoint.replaceAll("/+$", "") + "/v1/projects/" + project + "/regions/us-central1/jobs");
    var request = HttpRequest.newBuilder(url).timeout(Duration.ofSeconds(30)).GET().build();
    var client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
    var response = client.send(request, HttpResponse.BodyHandlers.ofString());
    if (response.statusCode() >= 400) {
      throw new IllegalStateException(response.statusCode() + ": " + response.body());
    }
    System.out.println(response.body());
  }
}
```

#### Run the example

```bash
java Hello.java
```

#### Verify the result

The request succeeds. An empty list or no printed rows is valid before matching resources exist. Continue with the documented workflows for a resource test.

gcloud CLI:

### gcloud CLI

### List submitted jobs

List submitted jobs; Dataproc runs Spark and Hadoop transformations over your datasets.

Use the Google Cloud CLI against the local gateway in its default open IAM mode. Run these exports in a terminal dedicated to local development.

Discovery check · reads existing local state.

#### Configure gcloud CLI

```bash
: "${SERVICE_USAGE_EMULATOR_HOST:?Export the LocalCloud environment first}"
export CLOUDSDK_CORE_PROJECT="${GOOGLE_CLOUD_PROJECT:?Export the LocalCloud environment first}"
export CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true
export CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAPROC="${SERVICE_USAGE_EMULATOR_HOST%/}/"
```

#### Run in your terminal

```bash
gcloud dataproc jobs list --region=us-central1
```

#### Verify the result

The request succeeds. An empty list or no printed rows is valid before matching resources exist. Continue with the documented workflows for a resource test.

Verify a resource workflow by creating isolated state, performing the operation, and reading the result back. Run cleanup even after a failure; delete only test-owned resources.

Connection failed? Check `localcloud status --verbose`, enabled services, and client exports. Stop accidental Google Cloud calls; correct local routing before retrying.

## Documented workflows

- ### Local cluster lifecycle

    Create a cluster in cluster mode with Docker access and wait for its lifecycle result. When testing local workload identity, select a service account and scopes and verify the identity seen inside the cluster.

- ### Spark/Hadoop execution in cluster and serverless modes

    Submit a small Spark or Hadoop job in cluster or serverless mode and inspect its terminal result and output. For identity-aware jobs, assert the account exposed by the local metadata relay.

## Service boundaries

- Requires Docker daemon access
- Cloud scaling, automated node repair, and production SLAs are cloud-only

## Code examples and integrations

- [Run Dataproc through local Docker](https://local.cloud/blog/run-dataproc-locally-docker/): Execution setup, job lifecycle, and Docker runtime boundaries.
- [Build an integration test loop](https://local.cloud/gcp-integration-testing/): Start the runtime, verify readiness, exercise your SDK, and clean up test state.

Agent testing

## Use Dataproc with an AI agent

Before the agent changes application logic, have it create local Dataproc state, read it back with the project SDK, and note which feature the check covered. That first local call catches accidental production routing and needs no GCP account, service-account key, or billing project.

### Configure this repo

```
Set up this repository to use LocalCloud for local GCP development. First read https://local.cloud/ai/agents.md, then inspect this repo, identify the GCP services and SDK language, configure emulator environment variables, and run the narrowest integration test against localhost. Do not use real GCP credentials or production endpoints.
```

Production endpoints, auth, IAM, quotas, regional behavior, and final release validation still need the target Google Cloud project.

[Agent resources](https://local.cloud/ai/) has more prompts and the AGENTS.md template.

## Related guides

- [Compatibility](https://local.cloud/compatibility/): compare all 27 service guides.
- [Configuration](https://local.cloud/docs/configuration/): service selection and runtime settings.
- [Official Dataproc documentation](https://docs.cloud.google.com/managed-spark/docs/concepts/clusters-overview): service concepts and cloud usage.