Parcel Forwarding by AGAS Service

Parcel Forwarding is an incremental feature added in the HPX system that allows the AGAS service itself to forward the parcels from the requesting locality to the destination locality. This feature would minimize the overall time taken for address resolution of a parcel’s destination and the parcel’s arrival at its destination.


In the previous implementation of AGAS, whenever there was a need for address resolution, the requesting locality would send a parcel to the AGAS server and the AGAS server would send a parcel back to the requesting locality with the resolved global virtual address (GVA). After receiving resolved address of the destination, the locality would then send the parcel to the final destination.

With Parcel Forwarding feature added,  the parcels could be sent directly to the AGAS server, and the server in turn would be able to forward the parcel to the final destination directly, hence minimizing the overall trip time of query-resolution-sending of the parcel.

The Application of choice is ShenEOS (shen equation of state), that maintains tables of nuclear matter at finite temperature and density with various electron fractions within the relativistic mean field(RMF), in a set of three dimensional data arrays enabling high precision interpolation of 19 relevant parameters required for neutron star simulation. It is implemented in HPX, with a HPX component encapsulating the non-overlapping partitioning and distribution of the ShenEOS tables to all available localities, thus reducing the required memory footprint per locality.

Test-bed for the experiment is a heterogeneous cluster with two 48 core SMP systems and 15 quad core Xeon type workstations.

For the test, an iterative operation on the ShenEOS tables of huge size is performed. The partitioning of data would allow exchange of parcels across localities, which serves the purpose of our test.


The test runs were performed with the local caching turned off so that more AGAS requests could be registered. The tests were performed for weak scaling with increasing resources as we increased work load.

The results of preliminary implementation of parcel forwarding shows slight speedup with parcel forwarding as compared with test run of the same application without parcel forwarding and every other test parameter remaining same. The results of the test runs are shown in the graphs below.


As a proof of concept, we can clearly see from the test results, that the ability of the AGAS service to forward parcel does minimize the round-trip time that is observed in the tests with parcel forwarding enabled. In essence, parcel forwarding cuts down the time spent to receive the parcel back to the requesting locality/node when an address resolution request is put to AGAS.

In the result, we observe an overhead with parcel forwarding enabled when there is less number of nodes involved. This overhead is due to extra cost of serialization and de-serialization of parcels at the action manager and the AGAS server side.

The current implementation does not consider the size of the parcel to be forwarded. Any parcel that needs address resolution is sent to the AGAS server, irrespective of the argument size. This may not be the optimal solution as there might be situations where parcel’s size is too big for it to: a) serialize and b) too huge to fit in memory of AGAS server. As an improvement to current implementation, an intelligent way of detecting parcel size beforehand and thereafter triggering a faster way of transferring parcels such as RDMA could be implemented.

GD Star Rating
    This entry was posted in AGAS, Performance by Vinay Amatya. Bookmark the permalink.

    About Vinay Amatya

    Vinay Amatya is a graduate student at Center for Computation, Louisiana State University. His interests are high performance computing, distributed computing, network discovery,graph problems. He has worked with HPX-Stellar group for over a year.

    Leave a Reply

    Your email address will not be published. Required fields are marked *