Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MINOR: R Devdocs CI fix #44929

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

MINOR: R Devdocs CI fix #44929

wants to merge 45 commits into from

Conversation

jonkeane
Copy link
Member

@jonkeane jonkeane commented Dec 4, 2024

Testing a fix for our dev docs job on releases. Currently based off of our maintenance branch to test — will rebase on main if this works.

kou and others added 30 commits October 10, 2024 11:32
…ath (apache#44359)

### Rationale for this change

This is not happen on Debian GNU/Linux stable and unstable. This is happen only on Debian GNU/Linux testing. So this may be a temporary problem.

### What changes are included in this PR?

Create a non-existent path manually as a workaround.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#44358

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…apache#44348)

### Rationale for this change

Other file systems such as GCS file system and S3 file system are enabled.

### What changes are included in this PR?

Enabled Azure file system.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#44347

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…tion in wheel-manylinux-*-cp313t-* (apache#44362)

### Rationale for this change

If interactive deb configuration is used, CI is blocked by waiting an user input.

### What changes are included in this PR?

Disable interactive deb configuration.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44355

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…row packaging (apache#44148)

## WIP

Creating the PR to validate if / what are the CI failures 

### Rationale for this change
Once the issue has been merged:
- apache#25118

We do not require numpy as a pyarrow dependency.

### What changes are included in this PR?

Remove numpy as a required dependency

### Are these changes tested?

Via CI

### Are there any user-facing changes?

Yes, pyarrow won't install numpy as a required dependency.
* GitHub Issue: apache#43846

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…d and test wheels on Python 3.13 (apache#44193)

### Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

* GitHub Issue: apache#43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…on on test_dataset.py (apache#44370)

### Rationale for this change

If pyarrow has been built without dataset test collection errors due to pytest.mark.parametrize requiring dataset.

### What changes are included in this PR?

Move dataset requirement outside of test collection by removing it from the parametrization.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: apache#44369

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
… to have macOS 12 pyarrow wheels (apache#44315)

### Rationale for this change

We did just bump MACOSX_DEPLOYMENT_TARGET from 10 to 12 for pyarrow wheels on 18.0.0, we probably should wait a little before dropping.

### What changes are included in this PR?

Revert moving macOS deployment target to 12

### Are these changes tested?

Will trigger wheel jobs on archery

### Are there any user-facing changes?

Yes, wheels will be available for macOS 12.
* GitHub Issue: apache#44314

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…OS (apache#44374)

### Rationale for this change

Java-jars are currently failing on macOS to install Python 3.13 because `python@ 3.13` (`python` is an alias of `python@ 3.13` now) isn't installed yet. Our link failure workaround is updating already installed `python@*` with `--overwrite` explicitly. `python@ 3.13` (`python`) isn't installed on GitHub Actions runner yet. So `brew bundle` installs `python` without `--overwrite`.

### What changes are included in this PR?

Install `python` with `--overwrite` explicitly before `brew bundle`.

### Are these changes tested?

Via CI

### Are there any user-facing changes?

No
* GitHub Issue: apache#44373

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…#44192)

### Rationale for this change

When adding Python 3.13 support (apache#43539), conda was not yet ready to support Python 3.13. But over the last weeks most conda packages having being updated, so trying again.

* GitHub Issue: apache#43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…verification jobs (apache#44382)

### Rationale for this change

Verification job is currently failing for Ruby.

### What changes are included in this PR?

We can't use Python 3.13 and `glib==2.80.5`. So we use old Python as a workaround.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: apache#44381

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…n verification when using Conda (apache#44388)

### Rationale for this change

Conda Python 3.13 fails when running verification tasks for Java. It's related to JPype.

See also:
* apacheGH-44389
* https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

### What changes are included in this PR?

Pin Python 3.12 on Conda integration jobs because JPype doesn't work with Python 3.13. We can use Python 3.13 again once JPype supports Python 3.13.

### Are these changes tested?

Yes via CI

### Are there any user-facing changes?

No
* GitHub Issue: apache#44386

Lead-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
### Rationale for this change

jemalloc may have a problem on ARM.
See also: apache#44342

### What changes are included in this PR?

* Disable jemalloc by default on ARM.
* Disable jemalloc for manylinux wheel for ARM.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* GitHub Issue: apache#44342

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…ase verification script (apache#44407)

### Rationale for this change

We can remove these pins now that conda-forge/glib-feedstock#191 has been closed. I've tested on my own crossbow and will test here momentarily. This effectively reverts apache#44270 and apache#44268.

### What changes are included in this PR?

Tweaks to release release verification script, specifically removing recently-added pins.

### Are these changes tested?

Yes: https://github.com/amoeba/crossbow/actions/runs/11335070956/job/31522511285.

### Are there any user-facing changes?

No.

Closes apache#44415.
* GitHub Issue: apache#44415

Authored-by: Bryce Mecum <[email protected]>
Signed-off-by: Bryce Mecum <[email protected]>
…est (apache#44425)

### Rationale for this change

It seems that the current `quay.io/centos/centos:stream9` for `linux/aarch64` is broken.

We use it for building and testing but testing is only broken for now. Because we use cached (a bit old) `quay.io/centos/centos:stream9` for now. Building will be failed when the cache is expired.

### What changes are included in this PR?

Change upload phase to after the build from after the test. Test is still failing but we can proceed our release process by this.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44422

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…#44390)

### Rationale for this change

See apache#44353 

`UnionMapWriter` does not override `map()` and falls back to implementation in `UnionListWriter` which causes exception when used.

### What changes are included in this PR?

Implement `map()` for `UnionMapWriter`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Changed behavior of the related function. 

* GitHub Issue: apache#44353

Authored-by: 0xderek <[email protected]>
Signed-off-by: David Li <[email protected]>
…tests (apache#44377)

### Rationale for this change

See apache#44361. This allows testing compatibility of the .NET Flight implementation with other Flight implementations.

### What changes are included in this PR?

* Adds a new `Apache.Arrow.Flight.IntegrationTest` project that can run in server or client mode for Flight integration tests.
* Includes the integration tests that send then retrieve data defined in JSON files, but doesn't add any of the named scenarios
* Configures archery to include C# in the Flight integration tests, but skip all the named scenarios
* Also skips tests that use dictionary data due to apache#38045, and the empty data test due to apache#44363

### Are these changes tested?

These changes are tests.

### Are there any user-facing changes?

No
* GitHub Issue: apache#44361

Authored-by: Adam Reeve <[email protected]>
Signed-off-by: Curt Hagenlocher <[email protected]>
…mplementation (apache#44424)

### Rationale for this change

See apache#44360

### What changes are included in this PR?

* Adds a new integration test to allow testing `do_exchange` between C++/Python and .NET.
* Updates the Flight stream reader to handle when a descriptor is sent in the first message without any schema.

### Are these changes tested?

* Yes, using the new integration test.

### Are there any user-facing changes?

No
* GitHub Issue: apache#44360

Authored-by: Adam Reeve <[email protected]>
Signed-off-by: Curt Hagenlocher <[email protected]>
…fy-rc-source-integration-linux-conda-latest-amd64 (apache#44395)

### Rationale for this change

Because JPype doesn't support Python 3.13 yet.

See also:
* apacheGH-44389
* https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

### What changes are included in this PR?

Pin Python 3.12.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44389

Lead-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
…e#44454)

### Rationale for this change

Executable scripts should have shebang.

### What changes are included in this PR?

Add `#!/usr/bin/env bash`.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44453

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
…44452)

### Rationale for this change

`dev/release/06-java-upload.sh` sometimes failed with the following HTTP error:

```text
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file (default-cli) on project standalone-pom: Failed to deploy artifacts: Could not transfer artifact org.apache.arrow:arrow-compression:pom:18.0.0 from/to apache.releases.https (https://repository.apache.org/service/local/staging/deploy/maven2): transfer failed for https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/arrow/arrow-compression/18.0.0/arrow-compression-18.0.0.pom, status: 408 Request Timeout -> [Help 1]
```

### What changes are included in this PR?

Use `-DretryFailedDeploymentCount=10` for retry on HTTP error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44451

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
…verifying binary wheels (apache#44460)

### Rationale for this change

Currently the folder used to verify the contents of the wheels is not correct when doing a release verification of the wheels and the verification fails. The script is supposed to check wheels individually and we are already verifying the contents on the wheels individually when generating the wheels.

### What changes are included in this PR?

Stop checking wheel content on release verification as the wheel content has already been validated when generating the wheels.

There is also a minor fix on the platform tags for the macOS wheels that was missed when we updated the MACOSX_DEPLOYMENT_TARGET.

### Are these changes tested?

Yes via archery.

### Are there any user-facing changes?

No
* GitHub Issue: apache#44459

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
…e#44450)

### Rationale for this change

Artifactory sometimes reports the following 503 HTTP error:

```text
BinaryTask::ArtifactoryClient::Error: failed to request: https://apache.jfrog.io/artifactory/arrow/debian-rc/pool/bookworm/main/a/apache-arrow/libparquet-glib1800_18.0.0-1_arm64.deb: PUT:  503
{
  "errors" : [ {
    "status" : 503,
    "message" : "503 : Failed during addStream to bucket"
  } ]
}
```

It seems that this is a temporary error. So we can retry on this type error for stable upload.

### What changes are included in this PR?

Retry on HTTP error.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.
* GitHub Issue: apache#44449

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
)

### Rationale for this change

Adopting apache#40226. 

 The creation and return of a shared_ptr does result in some performance overhead, that makes a difference for a performance-sensitive application.

If someone could use ChunkResolver to learn the indices, they could then instead access the data directly. 

### What changes are included in this PR?

- [X] Updates to documentation (thanks to @ SChakravorti21 )
- [X] Moving `ChunkResolver` to public API, and updating all references to it in the code

### Are these changes tested?

There seemed to be comprehensive tests already: https://github.com/apache/arrow/blob/main/cpp/src/arrow/chunked_array_test.cc#L324 If an edgecase is missing, I'd be happy to add it.

### Are there any user-facing changes?

`ChunkResolver` and `TypedChunkLocation` are now in the public API.
* GitHub Issue: apache#34535

Lead-authored-by: Anja Kefala <[email protected]>
Co-authored-by: anjakefala <[email protected]>
Co-authored-by: Bryce Mecum <[email protected]>
Co-authored-by: SChakravorti21 <[email protected]>
Co-authored-by: SChakravorti21<[email protected]>
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Felipe Oliveira Carvalho <[email protected]>
…tion (apache#44505)

### Rationale for this change

Since ChunkLocation is a typedef/alias, it needs to use the `doxygentypedef` Sphinx directive rather than `doxygenstruct` in order for it to show up in the generated HTML. Otherwise we get an error.

### What changes are included in this PR?

Just docs:

- Replace `doxygenstruct` directive with `doxygentypedef`
- Remove `:members:` directive as it's not needed

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes, though just docs.
* GitHub Issue: apache#44506

Authored-by: Bryce Mecum <[email protected]>
Signed-off-by: Bryce Mecum <[email protected]>
…h for cross-compiling (apache#44507)

### Rationale for this change

We can't use `find_package(gRPC)` and `gRPC::grpc_cpp_plugin` for cross-compiling because it's for host. We need `grpc_cpp_plugin` for target in cross-compiling.

### What changes are included in this PR?

Add `ARROW_GRPC_CPP_PLUGIN` CMake option that overwrites `gRPC::grpc_cpp_plugin` path found by `find_package(gRPC)`.

### Are these changes tested?

Yes.

conda-forge/arrow-cpp-feedstock#1432

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#44448

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
… verify-release-candidate-wheels.bat (apache#44462)

### Rationale for this change

The Windows wheel verification fails due to missing `PARQUET_TEST_DATA`

### What changes are included in this PR?

Add `PARQUET_TEST_DATA` to `verify-release-candidate-wheels.bat` which is only tested on the binary verification job.

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: apache#44461

Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
… version (apache#44613)

### Rationale for this change
Abseil's recommended usage is to build from trunk, this version does not provide a package version in CMake which breaks `resolve_dependency`.

### What changes are included in this PR?

Add `FindabslAlt.cmake` that adds a big version if it is missing to compare against our minimal version requirements.

### Are these changes tested?

CI

### Are there any user-facing changes?

Abseil build from head should now work without workaround.
* GitHub Issue: apache#44606

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
We skipped 18.0.0 and did our first 18.X CRAN release with 18.1.0 so we shouldn't have an 18.0.0 header.
…endencies on GHA (apache#43784)

### Rationale for this change

See apache#43547.

### What changes are included in this PR?

Adds two new new crossbow tasks for performing reverse dependency checking using https://github.com/r-devel/recheck: 

- `r-recheck-most` 
- `r-recheck-strong`

### Are these changes tested?

Yes. apache#44523 (comment).

### Are there any user-facing changes?

No.
* GitHub Issue: apache#43547

Fixes apache#43547.

Authored-by: Bryce Mecum <[email protected]>
Signed-off-by: Bryce Mecum <[email protected]>
@jonkeane
Copy link
Member Author

jonkeane commented Dec 4, 2024

@github-actions crossbow submit test-r-devdocs

@github-actions github-actions bot added the awaiting committer review Awaiting committer review label Dec 4, 2024
Copy link

github-actions bot commented Dec 4, 2024

Revision: 575cbdd

Submitted crossbow builds: ursacomputing/crossbow @ actions-3ab90fdd00

Task Status
test-r-devdocs GitHub Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants