Skip to content

Commit 45121a1

Browse files
nbhuiyannbhuiyan
committed
Fold MemorySegment primitive get/set into direct accesses on JDK 21+
MemorySegment.get and MemorySegment.set methods for byte, char, short, int, long, float and double data types dispatch through a deep VarHandle / MethodHandle / LambdaForm / VarHandleSegmentAsX / ScopedMemoryAccess chain. Even when such call chains inline fully, they leave a sequence of guards, such as LambdaForm dispatch, segment accessors and session checks, each of which cost a compare and branch per access and cannot be hoisted out of loops. This commit implements direct lowering of these accessors in J9RecognizedCallTransformer during its early pass (part of ILGen opts) to a guarded direct load or store at segment.min + offset whenever the layout argument is known at compile time. The per-access runtime overhead of the slow chain is replaced by compile-time folding of the layout's byte order and alignment constraint plus a single combined guard. The guard ORs together every condition the interpreted path checks and branches to the original call when any of them fails: - receiver is NativeMemorySegmentImpl or MappedMemorySegmentImpl (heap segments take the slow path) - 0 <= offset <= length - accessSize (bounds check) - min + offset aligned to the layout's byteAlignment, for alignment-constrained layouts (folded away for *_UNALIGNED layouts) - scope state is read through a volatile load, so the check is neither hoisted nor commoned across iterations - owner == null || owner == currentThread - !readOnly (for stores) The slow path is the unmodified original call, so each failing guard preserves the exact semantics and throws exceptions as necessary. Reversed-order layouts are handled by a compile-time endian conversion (a byte swap on the value, float and double go through their integer bit patterns). MemorySegment.get/set are given fine-grained recognized method ids so the transformation can match each primitive overload, and J9EstimateCodeSize keeps treating them as MemorySegment methods so the InterpreterEmulator's stateful bytecode iteration still folds accessHandle(). Two VM frontend queries, TR_J9VMBase::getLayoutByteOrder andgetLayoutByteAlignment, that read the known layout object's order and byteAlignment have been implemented for this transformation. Corresponding JITServer changes have also been implemented. This transformation requires additional work for AOT support, so it is currently skipped during AOT compilations. A new env option TR_disableFFMDirectLowering can be used to disable the transformation. Correctness of the fast path rests on three invariants: * the scope-state load is volatile * there is no yield point between that load and the access * native segment memory is freed only after ScopedMemoryAccess.closeScope0 has taken exclusive VM access A racing close therefore either lets an in-flight access complete before the thread gets to a yield point, or is observed as a closed state just prior the next access, which results in fallback to the slow path. This also covers Cleaner-managed implicit arenas without a reachabilityFence on the fast path. Signed-off-by: Nazim Bhuiyan <nubhuiyan@ibm.com>
1 parent 26b3d7d commit 45121a1

13 files changed

Lines changed: 839 additions & 7 deletions

‎runtime/compiler/codegen/J9RecognizedMethodsEnum.hpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -779,6 +779,15 @@ FirstJ9Method = LastOMRMethod + 1,
779779
jdk_internal_foreign_MemorySegmentImpl_unsafeGetBase, jdk_internal_foreign_MemorySegmentImpl_unsafeGetOffset,
780780
jdk_internal_foreign_MemorySegmentImpl_maxAlignMask,
781781

782+
java_lang_foreign_MemorySegment_get_OfBoolean, java_lang_foreign_MemorySegment_get_OfByte,
783+
java_lang_foreign_MemorySegment_get_OfChar, java_lang_foreign_MemorySegment_get_OfShort,
784+
java_lang_foreign_MemorySegment_get_OfInt, java_lang_foreign_MemorySegment_get_OfLong,
785+
java_lang_foreign_MemorySegment_get_OfFloat, java_lang_foreign_MemorySegment_get_OfDouble,
786+
java_lang_foreign_MemorySegment_set_OfBoolean, java_lang_foreign_MemorySegment_set_OfByte,
787+
java_lang_foreign_MemorySegment_set_OfChar, java_lang_foreign_MemorySegment_set_OfShort,
788+
java_lang_foreign_MemorySegment_set_OfInt, java_lang_foreign_MemorySegment_set_OfLong,
789+
java_lang_foreign_MemorySegment_set_OfFloat, java_lang_foreign_MemorySegment_set_OfDouble,
790+
782791
// Clone and Deep Copy
783792
java_lang_J9VMInternals_is32Bit, java_lang_J9VMInternals_isClassModifierPublic,
784793
java_lang_J9VMInternals_getArrayLengthAsObject, java_lang_J9VMInternals_rawNewInstance,

‎runtime/compiler/control/JITClientCompilationThread.cpp‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1094,6 +1094,16 @@ static bool handleResponse(JITServer::MessageType response, JITServer::ClientStr
10941094
vhObj = knot->getPointerLocation(vhIndex);
10951095
client->write(response, vhIndex, vhObj);
10961096
} break;
1097+
case MessageType::VM_getLayoutByteOrder: {
1098+
auto recv = client->getRecvData<TR::KnownObjectTable::Index>();
1099+
int32_t lbo = (int32_t)fe->getLayoutByteOrder(comp, std::get<0>(recv));
1100+
client->write(response, lbo);
1101+
} break;
1102+
case MessageType::VM_getLayoutByteAlignment: {
1103+
auto recv = client->getRecvData<TR::KnownObjectTable::Index>();
1104+
int64_t byteAlignment = fe->getLayoutByteAlignment(comp, std::get<0>(recv));
1105+
client->write(response, byteAlignment);
1106+
} break;
10971107
case MessageType::VM_getMethodAccessorIndex: {
10981108
auto recv = client->getRecvData<TR::KnownObjectTable::Index>();
10991109
TR::KnownObjectTable::Index maIndex = fe->getMethodAccessorIndex(comp, std::get<0>(recv));

‎runtime/compiler/env/VMJ9.cpp‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4785,6 +4785,68 @@ TR::KnownObjectTable::Index TR_J9VMBase::getLayoutVarHandle(TR::Compilation *com
47854785
return result;
47864786
}
47874787

4788+
// Returns the layout object pointer when layoutIndex is a known
4789+
// ValueLayouts$AbstractValueLayout instance, or 0 otherwise. The caller must hold VM access.
4790+
static uintptr_t knownAbstractValueLayout(TR_J9VMBase *fej9, TR::Compilation *comp,
4791+
TR::KnownObjectTable::Index layoutIndex)
4792+
{
4793+
TR::KnownObjectTable *knot = comp->getKnownObjectTable();
4794+
if (!knot || layoutIndex == TR::KnownObjectTable::UNKNOWN || knot->isNull(layoutIndex))
4795+
return 0;
4796+
4797+
const char * const layoutClassName = "jdk/internal/foreign/layout/ValueLayouts$AbstractValueLayout";
4798+
TR_OpaqueClassBlock *layoutClass = fej9->getSystemClassFromClassName(layoutClassName, (int)strlen(layoutClassName));
4799+
TR_OpaqueClassBlock *layoutObjClass = fej9->getObjectClassFromKnownObjectIndex(comp, layoutIndex);
4800+
if (layoutClass == NULL || fej9->isInstanceOf(layoutObjClass, layoutClass, true, true) != TR_yes)
4801+
return 0;
4802+
4803+
return knot->getPointer(layoutIndex);
4804+
}
4805+
4806+
TR_J9VMBase::LayoutByteOrder TR_J9VMBase::getLayoutByteOrder(TR::Compilation *comp,
4807+
TR::KnownObjectTable::Index layoutIndex)
4808+
{
4809+
TR::VMAccessCriticalSection getLayoutByteOrder(this);
4810+
uintptr_t layoutObj = knownAbstractValueLayout(this, comp, layoutIndex);
4811+
if (!layoutObj)
4812+
return LayoutByteOrder::UNKNOWN;
4813+
4814+
uintptr_t orderObj = getReferenceField(layoutObj, "order", "Ljava/nio/ByteOrder;");
4815+
if (!orderObj)
4816+
return LayoutByteOrder::UNKNOWN;
4817+
4818+
const char * const byteOrderClassName = "java/nio/ByteOrder";
4819+
const int byteOrderClassNameLen = (int)strlen(byteOrderClassName);
4820+
TR_OpaqueClassBlock *byteOrderClass = getSystemClassFromClassName(byteOrderClassName, byteOrderClassNameLen);
4821+
if (byteOrderClass == NULL)
4822+
return LayoutByteOrder::UNKNOWN;
4823+
4824+
void *leStaticAddr = getStaticFieldAddress(byteOrderClass, (unsigned char *)"LITTLE_ENDIAN", 13,
4825+
(unsigned char *)"Ljava/nio/ByteOrder;", 20);
4826+
if (leStaticAddr == NULL)
4827+
return LayoutByteOrder::UNKNOWN;
4828+
4829+
uintptr_t leObj = getStaticReferenceFieldAtAddress((uintptr_t)leStaticAddr);
4830+
bool layoutIsLE = (orderObj == leObj);
4831+
bool hostIsLE = TR::Compiler->target.cpu.isLittleEndian();
4832+
return (layoutIsLE == hostIsLE) ? LayoutByteOrder::NATIVE : LayoutByteOrder::REVERSED;
4833+
}
4834+
4835+
int64_t TR_J9VMBase::getLayoutByteAlignment(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex)
4836+
{
4837+
TR::VMAccessCriticalSection getLayoutByteAlignment(this);
4838+
uintptr_t layoutObj = knownAbstractValueLayout(this, comp, layoutIndex);
4839+
if (!layoutObj)
4840+
return 0;
4841+
4842+
// byteAlignment is declared on the AbstractLayout superclass and constrained to a
4843+
// power of two, and is >= 1 at construction time, so any other value means the read failed.
4844+
int64_t byteAlignment = getInt64Field(layoutObj, "byteAlignment");
4845+
if (byteAlignment < 1 || (byteAlignment & (byteAlignment - 1)) != 0)
4846+
return 0;
4847+
return byteAlignment;
4848+
}
4849+
47884850
TR::KnownObjectTable::Index TR_J9VMBase::getMethodAccessorIndex(TR::Compilation *comp,
47894851
TR::KnownObjectTable::Index methodIndex)
47904852
{

‎runtime/compiler/env/VMJ9.h‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -621,6 +621,43 @@ class TR_J9VMBase : public TR::FrontEnd {
621621
virtual TR::KnownObjectTable::Index getLayoutVarHandle(TR::Compilation *comp,
622622
TR::KnownObjectTable::Index layoutIndex);
623623

624+
/**
625+
* @brief Classification of a ValueLayout's byte order vs the host's native byte order.
626+
*/
627+
enum class LayoutByteOrder {
628+
NATIVE,
629+
REVERSED,
630+
UNKNOWN
631+
};
632+
633+
/**
634+
* @brief Determine whether a known ValueLayout's byte order matches the host's native byte order.
635+
*
636+
* The layout's order field (final, populated at construction time) holds a reference
637+
* to either ByteOrder.LITTLE_ENDIAN or ByteOrder.BIG_ENDIAN. We read the field and
638+
* compare to the LITTLE_ENDIAN static. The result lets the direct-lowering transform
639+
* decide whether to emit a byte-swap on the loaded value.
640+
*
641+
* @param comp the compilation
642+
* @param layoutIndex the ValueLayout$AbstractValueLayout known object index
643+
* @return NATIVE if layout.order == native byte order, REVERSED if it disagrees,
644+
* UNKNOWN if the layout object or required helper classes aren't loaded.
645+
*/
646+
virtual LayoutByteOrder getLayoutByteOrder(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex);
647+
648+
/**
649+
* @brief Determine a known ValueLayout's alignment constraint.
650+
*
651+
* The layout's byteAlignment field (final, a power of two >= 1 enforced at
652+
* construction time) decides whether an access through the layout has to be
653+
* alignment-checked.
654+
*
655+
* @param comp the compilation
656+
* @param layoutIndex the ValueLayout$AbstractValueLayout known object index
657+
* @return the byteAlignment value, or 0 if it cannot be determined.
658+
*/
659+
virtual int64_t getLayoutByteAlignment(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex);
660+
624661
/**
625662
* @brief Get the MethodAccessor Index of a java/lang/reflect/Method object, if the MethodAccessor has been set.
626663
* When the Method object is known, and its methodAccessor field is populated, we can evaluate the result of

‎runtime/compiler/env/VMJ9Server.cpp‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2417,6 +2417,23 @@ TR::KnownObjectTable::Index TR_J9ServerVM::getMethodAccessorIndex(TR::Compilatio
24172417

24182418
#endif /* defined(J9VM_OPT_OPENJDK_METHODHANDLE) */
24192419

2420+
TR_J9VMBase::LayoutByteOrder TR_J9ServerVM::getLayoutByteOrder(TR::Compilation *comp,
2421+
TR::KnownObjectTable::Index layoutIndex)
2422+
{
2423+
JITServer::ServerStream *stream = _compInfoPT->getMethodBeingCompiled()->_stream;
2424+
stream->write(JITServer::MessageType::VM_getLayoutByteOrder, layoutIndex);
2425+
auto recv = stream->read<int32_t>();
2426+
return (LayoutByteOrder)std::get<0>(recv);
2427+
}
2428+
2429+
int64_t TR_J9ServerVM::getLayoutByteAlignment(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex)
2430+
{
2431+
JITServer::ServerStream *stream = _compInfoPT->getMethodBeingCompiled()->_stream;
2432+
stream->write(JITServer::MessageType::VM_getLayoutByteAlignment, layoutIndex);
2433+
auto recv = stream->read<int64_t>();
2434+
return std::get<0>(recv);
2435+
}
2436+
24202437
TR::KnownObjectTable::Index TR_J9ServerVM::getMemberNameFieldKnotIndexFromMethodHandleKnotIndex(TR::Compilation *comp,
24212438
TR::KnownObjectTable::Index mhIndex, const char *fieldName)
24222439
{

‎runtime/compiler/env/VMJ9Server.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,8 @@ class TR_J9ServerVM : public TR_J9VM {
315315
virtual TR::KnownObjectTable::Index getMethodAccessorIndex(TR::Compilation *comp,
316316
TR::KnownObjectTable::Index methodIndex) override;
317317
#endif
318+
virtual LayoutByteOrder getLayoutByteOrder(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex) override;
319+
virtual int64_t getLayoutByteAlignment(TR::Compilation *comp, TR::KnownObjectTable::Index layoutIndex) override;
318320
virtual TR::KnownObjectTable::Index getMemberNameFieldKnotIndexFromMethodHandleKnotIndex(TR::Compilation *comp,
319321
TR::KnownObjectTable::Index mhIndex, const char *fieldName) override;
320322
virtual bool isMethodHandleExpectedType(TR::Compilation *comp, TR::KnownObjectTable::Index mhIndex,

‎runtime/compiler/env/j9method.cpp‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4557,7 +4557,47 @@ void TR_ResolvedJ9Method::construct()
45574557
setRecognizedMethodInfo(TR::java_lang_invoke_VarHandleSegmentAsX_method);
45584558
}
45594559
} else if ((classNameLen == 31) && !strncmp(className, "java/lang/foreign/MemorySegment", 31)) {
4560-
if (nameLen >= 3 && (!strncmp(name, "get", 3) || !strncmp(name, "set", 3)))
4560+
// Use a fine-grained recognizer for primitive data-type MemorySegment get/set signature matches.
4561+
// Otherwise, fallback to using the umbrella recognizer for MemorySegment accessors.
4562+
TR::RecognizedMethod specific = TR::unknownMethod;
4563+
if (nameLen == 3 && !strncmp(name, "get", 3)) {
4564+
if (sigLen == 42 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfByte;J)B", sigLen))
4565+
specific = TR::java_lang_foreign_MemorySegment_get_OfByte;
4566+
else if (sigLen == 43 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfShort;J)S", sigLen))
4567+
specific = TR::java_lang_foreign_MemorySegment_get_OfShort;
4568+
else if (sigLen == 42 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfChar;J)C", sigLen))
4569+
specific = TR::java_lang_foreign_MemorySegment_get_OfChar;
4570+
else if (sigLen == 41 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfInt;J)I", sigLen))
4571+
specific = TR::java_lang_foreign_MemorySegment_get_OfInt;
4572+
else if (sigLen == 42 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfLong;J)J", sigLen))
4573+
specific = TR::java_lang_foreign_MemorySegment_get_OfLong;
4574+
else if (sigLen == 43 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfFloat;J)F", sigLen))
4575+
specific = TR::java_lang_foreign_MemorySegment_get_OfFloat;
4576+
else if (sigLen == 44 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfDouble;J)D", sigLen))
4577+
specific = TR::java_lang_foreign_MemorySegment_get_OfDouble;
4578+
else if (sigLen == 45 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfBoolean;J)Z", sigLen))
4579+
specific = TR::java_lang_foreign_MemorySegment_get_OfBoolean;
4580+
} else if (nameLen == 3 && !strncmp(name, "set", 3)) {
4581+
if (sigLen == 43 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfByte;JB)V", sigLen))
4582+
specific = TR::java_lang_foreign_MemorySegment_set_OfByte;
4583+
else if (sigLen == 44 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfShort;JS)V", sigLen))
4584+
specific = TR::java_lang_foreign_MemorySegment_set_OfShort;
4585+
else if (sigLen == 43 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfChar;JC)V", sigLen))
4586+
specific = TR::java_lang_foreign_MemorySegment_set_OfChar;
4587+
else if (sigLen == 42 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfInt;JI)V", sigLen))
4588+
specific = TR::java_lang_foreign_MemorySegment_set_OfInt;
4589+
else if (sigLen == 43 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfLong;JJ)V", sigLen))
4590+
specific = TR::java_lang_foreign_MemorySegment_set_OfLong;
4591+
else if (sigLen == 44 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfFloat;JF)V", sigLen))
4592+
specific = TR::java_lang_foreign_MemorySegment_set_OfFloat;
4593+
else if (sigLen == 45 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfDouble;JD)V", sigLen))
4594+
specific = TR::java_lang_foreign_MemorySegment_set_OfDouble;
4595+
else if (sigLen == 46 && !strncmp(sig, "(Ljava/lang/foreign/ValueLayout$OfBoolean;JZ)V", sigLen))
4596+
specific = TR::java_lang_foreign_MemorySegment_set_OfBoolean;
4597+
}
4598+
if (specific != TR::unknownMethod)
4599+
setRecognizedMethodInfo(specific);
4600+
else if (nameLen >= 3 && (!strncmp(name, "get", 3) || !strncmp(name, "set", 3)))
45614601
setRecognizedMethodInfo(TR::java_lang_foreign_MemorySegment_method);
45624602
} else if (((classNameLen == 44) && !strncmp(className, "jdk/internal/foreign/NativeMemorySegmentImpl", 44))
45634603
|| ((classNameLen >= 42) && !strncmp(className, "jdk/internal/foreign/HeapMemorySegmentImpl", 42))) {

‎runtime/compiler/net/CommunicationStream.hpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ class CommunicationStream {
128128
// likely to lose an increment when merging/rebasing/etc.
129129
//
130130
static const uint8_t MAJOR_NUMBER = 1;
131-
static const uint16_t MINOR_NUMBER = 104; // ID: /HgmjAVrlw2DV8qPrQ3o
131+
static const uint16_t MINOR_NUMBER = 105; // ID: etwSOwaRRyEiXWkBxSiP
132132
static const uint8_t PATCH_NUMBER = 0;
133133
static uint32_t CONFIGURATION_FLAGS;
134134

‎runtime/compiler/net/MessageTypes.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ const char *messageNames[] = {
195195
"VM_mutableCallSiteEpoch",
196196
"VM_numInterfacesImplemented",
197197
"VM_getObjectClassInfoFromKnotIndex",
198+
"VM_getLayoutByteOrder",
199+
"VM_getLayoutByteAlignment",
198200
"CompInfo_isCompiled",
199201
"CompInfo_getPCIfCompiled",
200202
"CompInfo_getInvocationCount",

‎runtime/compiler/net/MessageTypes.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,8 @@ enum MessageType : uint16_t {
207207
VM_mutableCallSiteEpoch,
208208
VM_numInterfacesImplemented,
209209
VM_getObjectClassInfoFromKnotIndex,
210+
VM_getLayoutByteOrder,
211+
VM_getLayoutByteAlignment,
210212

211213
// For static TR::CompilationInfo methods
212214
CompInfo_isCompiled,

0 commit comments

Comments
 (0)